Comment by dotinvictim
1 day ago
local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable
1 day ago
local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
Beats Opus 4.6 at what exactly? It certainly isn't code.
I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o...
Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...
Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.
If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.
What configuration? What harness? These matter greatly to how a local model performs, in my experience.
> 5060
I think you're referring to a 5060Ti 16GB, yes?
32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.
Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.
But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.
In my own experience, qwen3.8-27b 4bit can consistently find bugs in software written by sonnet 5 and opus 5. But it does do that at maybe 1/10th the speed. Still a pretty good deal if you're coding without wanting to spend big.
qwen3.8-27b 4bit has a following specifically for being exceptionally gifted for such a small model.
It’s genuinely the first local model that has actually made me keep it around, it’s pretty good at spotting things existing tooling can’t, like comment rot/drift/summarising accurate git changes, it’s also the first local model that pushes back when it thinks it is right (I don’t dislike that even when it is wrong, it still hallucinates things that don’t actually work the way it confidently states they do).
It does clear the useful enough to be worth it bar though.
Zero interest in remote models but local ones if they offer utility, sure.
Runs pretty well on a 7900XTX as well.