Comment by rkangel
13 hours ago
It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years.
However there are other reasons (e.g. privacy) that might make it worth running locally for some people.
I think the biggest reason is to own the stack so your model can't be changed out from under you, but maybe I care about that too much.
> I think the biggest reason is to own the stack so your model can't be changed out from under you,
The concern would be future regulations that prohibit you from buying a hosted version of the model. Even that could be bypassed with a VPN to another country but it's more work to go through the payments.
As long as there is demand for a model, it will be hosted by multiple providers.
What if the model is hopelessly obsolete, and thus no demand, but I want that specific model? Owning the weights and hardware is not just solving for one problem. It eliminates all the classes of problems that occur outside of your building, if you have a solar and battery setup.
Also, on a more practical basis, what if the way it's served is bad. Maybe I want my specific KV setup, or ultra low quant for entertaining garbage at 200 tk/s
3 replies →
The only reason I'm considering picking one up is I think we're not that far away from compute limitations in consumer hardware.
I live in a place where using VPN is illegal and akin to "terrorism" because why would you want to hide what you are doing. Only bad guys hide. So if you use VPN, you are a bad guy.
https://srinagar.nic.in/notice/immediate-suspension-of-virtu...
Phones are randomly searched on the streets and if VPN is found, arrested
https://www.medianama.com/2026/01/223-jammu-kashmir-vpn-ban-...
https://timesofindia.indiatimes.com/india/after-vpn-ban-in-k...
“Out of the 15 individuals identified, five were minors who were counselled and advised in the presence of their guardians, with emphasis on awareness, lawful digital conduct, and the consequences of violating lawful orders,” he added.
You do, there's like 20 providers for any model on openrouter. You can also just spin bedrock or gcp and download the weights for later if you're worried. It's never going to make cost sense when the token rate is so low with how expensive ram is
What if the internet goes away?
And compliance.
I think the privacy argument that keeps coming up is overrepresented. Certainly ZDR is enough for an absolute majority of use cases? I see so much talk about local inference but I doubt most of it has privacy as a valid argument (not arguing it doesn't exist). It's fun to do things locally though. I've tried it as well but cloud is just faster and cheaper.
These companies have displayed zero respect for everyone's intellectual property getting these models trained.
I think not giving them your complete trust is reasonable! I'm not saying zero trust, and ZDR is fine for most things but I understand the people who don't want to stream their whole codebase out token by token.
Then use other providers hosting open models. Companies and individuals already put their whole code base on the cloud. I'm genuinely interested in privacy-oriented use cases where ZDR is not enough.
2 replies →
Privacy isn’t only, I don’t want anyone to have access to my data. It could also be, I don’t want anyone to know my use case because it’s niche and highly profitable.