Comment by kennywinker
1 day ago
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
1 day ago
It literally is usable now. A 5060 for $800 can run qwen3.8-27b 4bit at >40t/s, and the model beats opus 4.6 (max).
Beats Opus 4.6 at what exactly? It certainly isn't code.
I use a combination of a Claude Max subscription and local inference, including qwen3.8-27b, 4bit. I have found qwen to be absolutely useless at anything but very specific, surgical code changes. In my experience, for anything even remotely nuanced, a frontier model is required.
https://artificialanalysis.ai/?models=qwen3-8-27b%2Cclaude-o...
Index methodologies here: https://artificialanalysis.ai/evaluations/artificial-analysi...
Also see some specific benchmarks here: https://huggingface.co/Qwen/Qwen3.8-27B e.g. qwen scores 61.7 on swe bench pro, while opus 4.6 scores 53.4.
If you want to argue with the benchmarks, go for it. Fwiw i am not saying qwen3.8-27b is better or as good as the frontier. But i am saying it has crossed the threshold and is now a useful tool for coding and debugging. From my experience, Qwen3.6-35b-a3b was what you describe - it could do surgical edits only.
What configuration? What harness? These matter greatly to how a local model performs, in my experience.
> 5060
I think you're referring to a 5060Ti 16GB, yes?
32k context is easily done there. 64k can work with a more aggressive quant, but you lose a bit of speed.
Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.
But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.