Comment by jmyeet

There is built-in demand for local LLMs. An obvious example is law firms where using remote AI tools may be breaking privilege [1]. Any medical applications may likewise run into legal issues.

The problem is basically that we can't have nice things. AI chat logs themselves become another commodity to sell and to train on. We recently had a story about how Chinese firms are reselling Claude tokens [2]. The chat logs are a commodity here.

The only way to avoid this is to run LLMs locally. Even if you trust someone like Anthropic or Google, case law simply hasn't been established that the chat logs aren't discoverable.

Add to that that a sub-$5000 PC with a 5090 can already run a 31B model at reasonable inference speeds. Not amazing but good enough for many applications. Obviously that can't compete with Mythos but it doesn't have to. It also shows where the trend line is going for hardware. A $10k Nvidia GPU from 10 years ago now sells for scrap. What a consumer-level computer in 5 years can run locally will probably shock a lot of people.

[1]: https://news.ycombinator.com/item?id=48667495