Comment by PaulKeeble
13 hours ago
Locally the MOE based models get a lot more out of the hardware than a dense model. Qwen 3.8-27B is the current golden child if you have >24GB VRAM but because its a dense model it is pretty slow to run and requires a lot of hardware. A lot of the guides are very much focussed on getting the dense models to run but in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence, they behave a lot more like their peak size than the active parameter size.
RAM is going to be the big bottleneck for MOE (at least for now) as it will determine the quantisations you could run.
On a 3080 10GB and 32GB of RAM I can run the qwen-next coder model and that is a 127B model and it gets 40-50 Token/s with 65k context. That is a lot more usable than a Quantised Qwen3.8-27B in my experience both because its faster and more capable.
I am looking forward to Qwen4 soon, I am hoping we get some more 27-35B MOE models, those are going to be really good for local hardware.
> in practice the MOE models with few active parameters are just faster and easier to run without too much loss of intelligence
Is this true for coding tasks? Genuinely asking because I don't yet have the hardware to be able to run/compare. I thought I remembered the sentiment that the 27B dense was better than the 35B MoE
The dense is better in its result, but is out of reach for most hardware compared to the MoE models, and the MoE models produce output tokens much much faster.