Comment by canpan
10 hours ago
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.
Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But for coding, the local models are already good enough for a vast amount of use cases, if you are an experienced developer yourself and can guide it.
I think the future should be efficient quantization which can make model with the same level as Qwen 3.8 27b now accessible from our laptop.
This is mostly a memory bandwidth issue. Qwen 3.8 27b and the related Bonsai 2 27B run on my M5 Max laptop with 36GB of ram. It’s a little slow due to my specific model having lower memory bandwidth than higher end models with more ram and cores. MoE models perform better here and expert streaming engines can allow larger than memory models to run.
Multiple teams and efforts are now making Qwen 3.8 27b work much faster on Mac and CUDA and the inference speed is doubling almost every week. There's now ongoing competition about who will make it work faster with exactly the same byte output at https://mlx.fast . TensorFold is the project to follow.
1 reply →
The future is already here :)
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
2 replies →
I don't know, it hallucinates quite badly - confuses js with python.
1 reply →
Its possible (though very hard to do well, and needs a fair bit of compute) to improve the local models we have, too. Continued training is possible as time goes on.
>The nice thing is that current local modes don't go away.
Given pace of advance I'm not convinced this is a winning strategy. The models of today (open or not) are way smarter than those of a year ago.
...so we really do need open models to keep pace, not just be statically competitive at a given point in time
Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that. My secret is that I have this relatively unknown technique for solving problems where I use this quantum engine that developed from just atoms over 4.54 billion years into the most sophisticated machine in the known universe. It's a frontier model that every human has access too, actually.
So when I really want to solve a problem, I just think on it. And when I need something to iterate through the things I thought of quickly, I use deepseek.
> Even if Deepseek v4.1 is the last open source LLM I have access to, I would be perfectly fine with that.
Your quantum engine aside, current open-source LLMs will not age well. They have a knowledge cutoff. If you ask them five years from now about something that didn't exist when they were trained, chances are they won't be able to provide a useful answer. They may still be useful for many applications, but their value will eventually deteriorate. Having continuous access to up-to-date open-weight LLMs is therefore about much more than simply having access to "better" models.
2 replies →
Yep, fully satisfied with DS 4.1 Flash or GLM 5.3 Flash. I prefer using one chat at a time, actually thinking through things with it, as opposed to the YOLO run all night multi-agent thing. I mean I’m still okay with subagents if my main thread needs to explore a few options and then report back, but all on the same idea.
2 replies →
The thing here for me is not that I’m unable to solve problems with my brain, it’s that I value the automation in labor provided by coding agents. I do not want future advancements to be locked behind proprietary doors.
2 replies →
You say that now, but you don't know what the future models will be capable of. If models get ridiculously good, you might not be perfectly fine with that.
1 reply →
How do you run DS 4.1 Flash if your computer isnt powerful and everything?
3 replies →
The fact that you point to the latest model as „I’d be fine with that“ proves my point.
If expectations were static you’d be fine with qwen 2.5…but nobody sane says that today.
It’s a moving target thus we need open weights to move
1 reply →
I’m all for local models and getting away from Anthropic/OAI but Qwen 27B might as well be a horse drawn carriage compared to a 2026 Porsche. Like, I guess it’s useful if you had nothing better but acting like it’s comparable to anything the frontier models have is just disingenuous.
disagree, it’s absolutely comparable. but an m5 max is where it shines. and there are 50 flags to finetune with llama server before you get it right. i do prefer llama server over mlx currently on mac as well.
It’s so poor for most tasks that you’re faster doing it manually.
DeepSeek and GLM are useful, though. Even the flash variants.
3 replies →