Comment by pizza234
1 day ago
> Having a machine that can run some modest local LLMs, like the Gemma 4 12B, is really worth it.
Cloud models are (much) faster, they don't consume so much power/generate heat, they have much bigger (LLM) context, they're much more precise and they have a much wider (engineering) context of the given problem.
Except privacy and use cases that are blocked by cloud models (e.g. reverse engineering), local LLMs are currently an expensive toy.
When I try to program with a local LLM (I'm on a 32/128 GB system), I end up wasting time compared to a cloud LLM.
Again, I would not argue against any of this.
And I can't say that I won't switch to openrouter (even just for the same models) at some point.
But one of the things I have found about my own process learning is that some lessons only come to you when you make yourself available to them. And if that means doing things the difficult way, that is what you should do.
Difficult... and wastefully expensive
Seems like an investment into building expertise, which is likely to have high ROI in the future, rather than a wasteful cost.
I mean, it's a (secondhand) computer I bought for other tasks (processing very large photos, compiling large apps quickly). It's running all the time. It can also run LLMs when I want to.
The rest of my life is ultra-frugal so I am relaxed about this.
3 replies →
People pay thousands for model trains, everyone needs a hobby.
3 replies →
From your post I can only perceive the instinct to pick a side, and trying to make sure it is the "winning side". But the truth is far more nuanced. I have acces to both, paid and local models, and even if slower, the local models have been far more educative about how these technologies are put together, and what is required for local computing to thrive again. Paid models will not suddenly disappear just because I play with glm-4.6 on Ollama. At the same time, my work pays the cloud subscription and I use the cloud models to perform the tasks my work requires. There's no need to choose one side.
> currently
The interesting question is whether that gap will narrow, and if so, how much, and on what timescale.
The exact answer to this question is not knowable, but if you are the kind of person who comes to a site called "hacker news", and you think there is a nonzero chance that the answer is that yes, the gap will narrow and this won't always be an expensive toy, then now seems like a pretty great time to get in the game and start exploring the capabilities.
I agree completely. I think local AI is best limited to purpose built SLMs; all this craze around running quantized coding LLMs has taken the attention off SLMs.
Same. Local LLMs are fun to experiment with, but when I want generated code of a sufficient quality, I use a cloud LLM.
The key word there is 'currently'.
Economies of scale are a fact of nature and aren’t going to be subverted in the future by even the most advanced local models
Which is of course why, if you want to render 3d scenes to play a video game, you have to rent time on a mainframe system. I don’t see that changing ever - it’s just economies of scale!
(sarcasm, btw)
The economies of scale gains are lost because you still have a middle man hosting provider who wants to profit too.
Over the long term it's always been better to buy than to rent, even if the renting option is technically more efficient on the GPUs, you don't have to pay some hosting providers profit margin.
2 replies →
Things can get both more expensive and cheaper at scale, hence the term.
For example (and relevant to AI) I can generate electricity on my roof at $0.20-25/kWh, batteries included. In California the electric utility can’t offer it cheaper than $0.30-0.50/kWh. Therefore at scale, electricity is actually more expensive.
There are many such examples.
2 replies →
Setting aside that very little about economics rises to the level of "facts of nature" like physics...
What makes you so certain that economies of scale won't work the opposite way you imagine? E.g., if model improvement tapers off, but RAM costs decline (hard to believe atm, but historically likely), then eventually everyone will be able to run SOTA models on their personal hardware.
Heck, even if model sizes simply grow more slowly than RAM costs decrease, the same would happen.
... said the IBM executive to a young Bill Gates.
> Cloud models […] don't consume so much power/generate heat
I do realize the cloud is just someone else’s computer right? Power goes in, tokens and heat come out - just in another place
The cloud computers produce more tokens per watt. That said, if you have a computer at home running 24/7 for other reasons and you also can use it for some LLM work, why not.
Anything done local will likely come at higher cost and at scale with less energy efficiency and commodity, with less possibility to fine tune engineer deeply on wider horizon of issues.
That's never the point of keeping local alternatives though.
Right.
For me this dates all the way back to installing Slackware 1.0 (0.99pl12!) on an offline 486SX rather than just using the internet-connected workstations in the lab.
Here, I already had a Mac that was powerful enough to run a local LLM, so now I do, because I can.