Comment by khurs
2 months ago
>In the long run I truly believe local AI will win
What do you mean by 'win'?
For a normal coder/person's use cases, yes. But AI companies are becoming more specialised in different fields and these tailored models will be leagues ahead in those niches.
The way I see it - Opus 4.8 xhigh can do any programming task with a programmer instructing it. If Apple releases local model together with a device that can run said model it would render OpenAI/Anthropic useless for vast majority of usecases.
And if a local mcahine can run something like Opus 4.8, who is to say that those "specialized" models would just not come at a later date, or even loading open models wouldn't be an option with something like M7-verified flag from huggingface that would make it extremely easy for any consumer to just play around.
But most private buyer and most business buyers don't get anything more than base models.
As of yet no indication that small models that can fit in 8gb/16gb can be fully relied upon?
There is built-in demand for local LLMs. An obvious example is law firms where using remote AI tools may be breaking privilege [1]. Any medical applications may likewise run into legal issues.
The problem is basically that we can't have nice things. AI chat logs themselves become another commodity to sell and to train on. We recently had a story about how Chinese firms are reselling Claude tokens [2]. The chat logs are a commodity here.
The only way to avoid this is to run LLMs locally. Even if you trust someone like Anthropic or Google, case law simply hasn't been established that the chat logs aren't discoverable.
Add to that that a sub-$5000 PC with a 5090 can already run a 31B model at reasonable inference speeds. Not amazing but good enough for many applications. Obviously that can't compete with Mythos but it doesn't have to. It also shows where the trend line is going for hardware. A $10k Nvidia GPU from 10 years ago now sells for scrap. What a consumer-level computer in 5 years can run locally will probably shock a lot of people.
[1]: https://news.ycombinator.com/item?id=48667495