Comment by KronisLV
2 days ago
I hope really badly that we'll get a new 35B A3B or similar MoE model!
I also miss the Qwen 3 Coder Next, which was 80B A3B, there are quite a few use cases where a non-dense model <100B would be the sweet spot (when you have the VRAM but not the TDP or compute power). Heck, I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Also alternate link for viewing the images without signing in: https://xcancel.com/Alibaba_Qwen/status/2088280182356611304
I'm hoping too that they'll put out some MoE variants.
Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.
Edit: Like its predecessors, 3.8 seems really inclined to overthinking, and on a 27b dense model that's kind of painful. I think I'm going to stick with gemma4:26b-a3b as my go-to because it runs about 4x as fast and tends to only need a fraction of the tokens in its 'thinking' stage to get the same or similar answer.
Reduce or turn down thinking:
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Yeah, that's probably the answer given that it apparently defaults to 'xhigh'.
2 replies →
[flagged]
"inclined to overthinking," holy cats you're not kidding! On a Mac mini M4 Pro 64GB I prompted it with "svg owl" and it thought for 17m12s, outputting 36.3KiB of thinking chatter. It did end up producing a 20.2KiB HTML+JS+SVG file with a very nice owl, including cursor-tracking animation, but it ran for more than a half hour!
The MoE models are stupendously faster.
I think that's called test-time scaling i.e using more tokens at infer time to squeeze out higher model performance. That's must be part of the explanation for good benchmark results.
2 replies →
what harness did you use?
1 reply →
Did you try the claude reasoning traces finetune for qwen3.6? I find that it works muuch better. I assume the same 3.8 finetune will be released at some pointas well.
Edit: link - https://huggingface.co/rico03/Qwen3.6-27B-Claude-Opus-Reason...
Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.
[self-reply because comment edit window ended]: I now have a head-to-head benchmark. On my wheezy laptop (specs in sibling comment), this new dense model, Qwen 3.8 27B, gets ~4 tokens/second on generation. The older mixture-of-experts model, Qwen 3.6-35B-A3B, gets ~20 tokens/s.
MoE is literally 5x times faster (on CPU) than comparable dense Qwen!
Tested:
Qwen3.6-35B-A3B-UD-Q4_K_XL
Qwen3.8-27B-UD-Q4_K_XL
35A3 might be more comparable to 10 dense.
27 dense is far more capable than 35A3.
1 reply →
If you are not already using MTP, you should be able to get ~2x decode tokens/s with Qwen 3.8 27B.
yeah, that's the A3B part; going up to A5B would probably also feel comfortable.
on the 395+ AI MAX w/128GB, the A10B qwen 3.5 can do a lot of long running work if you don't need to baby sit it. deer-flow works well like that.
Got Qwen3.8 to run on my Framework 12 Intel Core 13 Gen Raptor Lake i5-1334U small laptop with 48G RAM stick:
llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL
but it failed my basic prompt to compose a vim regex to match CamelCaseWords
downgraded a bit with Q4_K_M from ollama run qwen3.8:27b
and /set nothink and at least 1 regex matched FooBar
prompt eval at 2.8 t/s eval at 0.94t/s
I particularly enjoyed this usage of the regex: /%\1\%/ ... onward for 700+ characters of \%\/ :-)
Mind sharing your laptops specs? Just interested to see what is needed to locally run Qwen3.6-35B-A3B
Don't mind! It's 64 GiB dual-channel DDR5-6400, i.e. roughly 100 GiB/s of bandwidth. (AMD 7840U (Zen 4))
I'm using a Q4 quantization from unsloth (Qwen3.6-35B-A3B-UD-Q4_K_XL). It gets up to ~20 tokens/second in generation. I don't know precisely how much KV cache I can safely use, but it's in between 140k–256k. (I.e., 140k reliably works, 256k kernel-crashes from OOM. Don't feel like bisecting).
Inference is llama.cpp with the Vulkan GPU backend on Linux. (I.e., -DGGML_VULKAN=1 on the llama.cpp build, and --gpu-layers all on llama-cli or llama-server. (And for my specific setup, two kernel parameters specific to amdgpu: ttm.pages_limit and ttm.page_pool_size. A driver VRAM limiter. Look it up if you're on amdgpu!)).
2 replies →
AgentWorld is a pretty recent MoE release from the Qwen folks with quite a bit better performance than 3.6, released around the same time as 3.7 Plus/Max, I guess. Not sure why it didn't get more attention, as it is definitely better than 3.6 35B A3B on all dimensions, but especially for agentic use. Still nowhere near the dense models (even 3.6 27B), but clearly an upgrade in the small Qwen MoE line.
https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B
KAT Coder is another Qwen 3.6 MoE fine-tune that also improves on 3.6 by a measurable amount.
But, I do hope for a bigger MoE, in the 70B to 120B range, something in the Coder Next lineage. I've got a Strix Halo that isn't getting used to its best ability because the best models all run fine on my faster desktop dual 32GB GPU setup, and they run too slow for comfort on the Strix Halo.
HUH? AgentWorld is a simulation of the world (e.g. tools and programs) for use by an agent!
And, yet it outperforms Qwen 3.6 35B A3B on Terminal Bench and SWE, etc. I dunno.
Edit: I guess you're right; apparently it's for simulation. I didn't look into it beyond the benchmarks. But, it does work in an agentic context, regardless. It'll write code, and drive an agent.
Me too. 35B A3B runs really fast on my MacBook Pro (M4 Max) and is suitable for real-time tasks like dictation post-processing. The dense model is not.
I'm still confused about Qwen 3.6 35B A3B. Everything I read said that the 27B model performs better at coding tasks, so what's the purpose of the 35B model?
It's better for VRAM poor people. I get 4-5 t/s with 27B and 20-30 t/s with 35B A3B.
Also radically better on an M1 Max. I get well up into the 60s t/s with the A3B, stuck at 9.5 or so with this new 27B, though perhaps an MLX build will help.
The "a3b" refers to its active parameters -- unlike 27b it is a mixture of experts model, so it runs much faster, about as fast as a 3b model, but needs as much memory as a 35b model! So good for unified memory systems like macs :)
MoE models have less active parameters in play at any particular moment, so they perform much faster on lower bandwidth memories (like Strix Halo or DGX Spark) and also use less memory generally.
Dedicated GPU memory tends to be much faster (either DDR6 or HBM). So if you can fit a whole dense model in it, you're probably better off with that.
anecdotes: 35B-A3B does want more memory, bigger model. But if you get it running it will be faster and more enjoyable to use -- text will fly by -- due to only 3B params being active, in my experience at least.
Runs faster on bad hardware.
Or even on fairly high end (by general public standards, not gaming community standards) consumer hardware. Fantastic fit for M-series Macs with 32-96GB RAM.
35Ba3b is usable in plain cpu inference on a fast server, the dense model is MUCH slower. On GPU the 35ba3b is still around 2x the tok/s single threaded, which can be a good tradeoff for some applications.
> I'd gladly take A5B or A8B or even A10B as a sort of middle ground.
Whats up with focusing on the active param count? Do yall fiddle with the weights or something?
Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s
Thank you Kenny
You can run these on CPUs at a somewhat reasonable speed.
Or (somewhat) low TDP GPUs for that matter, like workstation ones, that might have enough total VRAM but not the best bandwidth/compute.