Comment by colordrops
2 days ago
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
2 days ago
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
It's better for VRAM poor people. I get 4-5 t/s with 27B and 20-30 t/s with 35B A3B.
Also radically better on an M1 Max. I get well up into the 60s t/s with the A3B, stuck at 9.5 or so with this new 27B, though perhaps an MLX build will help.
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)
MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.
Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.
anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.
Runs faster on bad hardware.
Or even on fairly high end (by general public standards, not gaming community standards) consumer hardware. Fantastic fit for M-series Macs with 32-96GB RAM.
35Ba3b is usable in plain cpu inference on a fast server, the dense model is MUCH slower. On GPU the 35ba3b is still around 2x the tok/s single threaded, which can be a good tradeoff for some applications.