Comment by rafaelmn
2 hours ago
Except there's a huge gulf of self-hosting and using API hosts - no way you can reach the economics of a shared host. Privacy is a problem but you can chose who you host with and where it's hosted (which jurisdiction).
When privacy/compliance really starts to matter it's up to the client/business to provide you with tooling - you're not running that on your own hardware anyway.
So the local AI for individuals is just a hobby/gimmick at this point not a rational decision. Self-hosting for business is a different story.
I'm not sure. The problem with the cloud llm's is they are complete black boxes that change frequently and randomly day by day.
If you run Qwen 3.8 on your own hardware, every single day, it's the exact same model running in the exact same way.
Yes, it's nowhere near as "smart" as the cloud based models. But it's consistent.
So the workflows and "ways of working" you create will work mostly similar day to day.
With Claude/OpenAI you frequently find days where the models are useless, and days when they are out of this world.
So I guess the choice comes down to:
1. Randomly the smartest thing on the planet with unpredictable rate limits that is mostly amazing, but frequently messes with your workflows
2. A really good local coding model that is consistent every day with no rate limits
I'm not sure. My gut feeling is maybe the right answer is a mix of both.
Gambling on the biggest models, hoping they are working smart that day, when planning or doing very complex work. Then doing most of the tasks/daily work using local models??
You can run any open model on a shared API host via OpenRouter and pin to which host you want to go for the quant/privacy/etc. mix you care about. You can pay them directly if you don't want the OpenRouter overhead - but the convenience of switching, having one invoice, etc. is worth it IMO
It's not closed hosted models vs open local models, it's hosted open models vs local open models where the math doesn't work for local LLMs.
The only local inference use-case I can think of is porn generation (because most providers don't want to deal with it) and illegal shit like hacking to minimize the tracing.
And if you're super paranoid - but honestly giving sensitive info to LLMs in any scenario is a gamble.
If you game and can use your GPU I guess then it works as well but models that fit into a gaming GPU suck too much to bother IMO.