← Back to context

Comment by ashwinsundar

7 days ago

I just go outside when my internet is down for 15 minutes a year. Or tether to my cell phone plan if the need is urgent.

I don't see the point of a local AI stack, outside of privacy or some ethical concerns (which a local stack doesn't solve anyway imo). I also *only* have 24GB of RAM on my laptop, which it sounds like isn't enough to run any of the best models. Am I missing something by not upgrading and running a high-performance LLM on my machine?

8 comments

ashwinsundar

Reply
filchermcurr  7 days ago

I would say cost is a factor. Maybe not for OP, but many people aren't able to spend $135 a month on AI services.

  • ashwinsundar  7 days ago

    Does the cost of a new computer not get factored in? I think I would need to spend $2000+ to run a decent model locally, and even then I can only run open source models

    Not to mention, running a giant model locally for hours a day is sure to shorten the lifespan of the machine…

    • outworlder  7 days ago

      > Not to mention, running a giant model locally for hours a day is sure to shorten the lifespan of the machine…

      That is not a thing. Unless there's something wrong (badly managed thermals, an undersized PSU at the limit of its capacity, dusty unfiltered air clogging fans, aggressive overclocking), that's what your computer is built for.

      Sure, over a couple of decades there's more electromigration than would otherwise have happened at idle temps. But that's pretty much it.

      > I think I would need to spend $2000+ to run a decent model locally

      Not really. Repurpose second hand parts and you can do it for 1/4 of that cost. It can also be a server and do other things when you aren't running models.

    • haiku2077  7 days ago

      The computer is a general purpose tool, though. You can play games, edit video and images, and self-host a movie/TV collection with real time transcoding with the same hardware. Many people have powerful PCs for playing games and running professional creative software already.

      There's no reason running a model would shorten a machine's lifespan. PSUs, CPUs, motherboards, GPUs and RAM will all be long obsolete before they wear out even under full load. At worst you might have to swap thermal paste/pads a couple of years sooner. (A tube of paste is like, ten bucks.)