Having heavily evaluated both antirez’s ds4 flash and Qwen 3.6 27B at FP8 and Q8: it depends. The quantised Flash is better in a number of tasks despite running much slower on my DGX Spark-alike.
27B is amazing for its size but has some surprising limits when used for longer agentic coding sessions, especially if you’re using tools that are outside the stock standard web tech stuff: it really isn’t good at Relay, for example.
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work).
Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.
I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the draft count low, acceptance rate is terrible beyond half a dozen and it requires more memory
Wow thanks for pointing this out! This is actually what I was hoping would happen when the deepspec stuff dropped! And having zlab create these confirms my bias that I think open models are the way.
Isn't Q8 way overkill these days? I see many graphs showing Q4 or Q5 having less than %1 deviation. Nvidia's NVFP4 Qwen quantization should be even better due to its better training methods.
Q8 isn't overkill if you have sufficient RAM to fit the whole model, and you care about quality. There's a number of people who have enough hardware to fit exactly one 27B to 35B size Q8 model and not more than that, so if you can fit the whole thing in Q8, no reason to use Q4 or Q6.
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.
And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).
Qwen3.6 below Q8 often can't exit a reasoning loop (until it hits max output token count), forgets to insert a tool call, often mistakenly inserts them inside the thinking block... It's still usable though.
Careful with those graphs, they're usually evaluating the model on KLD on relatively short transcripts. When you're running with 100k token contexts and the model running close loop a difference that looks small in terms of KLD may be quite substantial.
I'm not aware of any great benchmarks that work by giving it a live agentic harness and a number of realistic tasks that take most of the context window to accomplish and evaluate success rate and tokens to completion... but that's what you'd really want to use to judge different quantization levels.
I'm dusting off the local models thing after being too busy for a while and it is crazy how wrong the benchmarks and general narrative is.
First, vLLM is like, we can do better than this, we need a better default target. It parses capability wrong, silently falls back on sm_80 xmma kernels when cutlass3x_sm120_bstensorprop_... CASK kernels are available, has mad Python in the hot path, emits dubious tool call syntax, and just managed to do something so off-road it tripped a driver bug that managed to wedge MMIO so bad the EC couldn't get an SBR out so the fan was going literally max until I hard pulled power. They say AMD is worse and I'll take their word for it because patching the driver and Inductor both in the same day is plenty of grief for me.
But the amazing thing and why your comment prompted me to write all this is that Qwen3.6 is insanely good, so good I am seriously questioning the MoE dogma. I was running the NVFP4 quant with the BF16 drafter, 100+ tokens a second of clean, legible reasoning trace and flawless tool calls. It's small and you can tell it hasn't memorized half the internet, but it's reasoning is like, better than most frontier. Opus has cleaner reasoning, GPT 5.x and Gemini 3.x Pro do not. If someone scaled that boi up by 5x? I get the feeling DeepSeek did so many arch innovations in one release that they just didn't quite have the convergence, this is like, the fundamentals as artistry. It's wayyyyyy stronger than GPT-4o at over a trillion parameters.
The other thing is I was using it on OpenRouter and it was all janky in the traces, stuttering and going in circles. On another day I would have been like "what do you expect it's the size of an iPhone". I wonder how many other people have drawn that conclusion too.
I'm not going to call it a conspiracy because it's explained by neglect, but we haven't even scratched the surface of the local model ceiling. With a harness that kept the data fresh, scale up the parameter count a bit, stretch context out a bit, and write an LLM serving engine that isn't hobbyist Python jank wrapped around fuckin inductor/triton jank?
That's Claude Code Opus experience on an expensive gaming box.
Having heavily evaluated both antirez’s ds4 flash and Qwen 3.6 27B at FP8 and Q8: it depends. The quantised Flash is better in a number of tasks despite running much slower on my DGX Spark-alike.
27B is amazing for its size but has some surprising limits when used for longer agentic coding sessions, especially if you’re using tools that are outside the stock standard web tech stuff: it really isn’t good at Relay, for example.
I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as corroborated by antirez's work).
Its also only 13B active, so your decode speed would be nearly 2x that of Qwen3.6-27B. So there are other latent benefits as well.
z-lab has been dropping dflash addons for a lot of models
https://huggingface.co/collections/z-lab/dflash
I'm running the qwen3.6-27B + dflash on a spark and tgen is way up, but keep the draft count low, acceptance rate is terrible beyond half a dozen and it requires more memory
Wow thanks for pointing this out! This is actually what I was hoping would happen when the deepspec stuff dropped! And having zlab create these confirms my bias that I think open models are the way.
For most coding or agentic tasks, Qwen 3.6 27B likely outperforms, yes.
For 'general intelligence', DS4 Flash seems to be a noticeable step up still.
I suspect it would depend on the task. DS4-flash does, as previously mentioned, handle quantization very well. Even at 2-bit it's still very coherent.
Isn't Q8 way overkill these days? I see many graphs showing Q4 or Q5 having less than %1 deviation. Nvidia's NVFP4 Qwen quantization should be even better due to its better training methods.
Q8 isn't overkill if you have sufficient RAM to fit the whole model, and you care about quality. There's a number of people who have enough hardware to fit exactly one 27B to 35B size Q8 model and not more than that, so if you can fit the whole thing in Q8, no reason to use Q4 or Q6.
When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its not a good proxy for whats actually degradaing at the task level across an entire rollout.
And for MoEs, very small amounts of loss can mean you're flipped to entirely different experts (this is also a problem more broadly with numerical stability issues too).
Qwen3.6 below Q8 often can't exit a reasoning loop (until it hits max output token count), forgets to insert a tool call, often mistakenly inserts them inside the thinking block... It's still usable though.
Careful with those graphs, they're usually evaluating the model on KLD on relatively short transcripts. When you're running with 100k token contexts and the model running close loop a difference that looks small in terms of KLD may be quite substantial.
I'm not aware of any great benchmarks that work by giving it a live agentic harness and a number of realistic tasks that take most of the context window to accomplish and evaluate success rate and tokens to completion... but that's what you'd really want to use to judge different quantization levels.
It depends on model size I think, but yeah, from my understanding at ~30B and below Q6 or even Q4 will get you 95%+ of the way there
1.01 over 30k tokens is over a googol (a large number with 100 zeroes)
qwen 27b at q8 is slower and worse than ds4 at q2 in my experience.
I'm dusting off the local models thing after being too busy for a while and it is crazy how wrong the benchmarks and general narrative is.
First, vLLM is like, we can do better than this, we need a better default target. It parses capability wrong, silently falls back on sm_80 xmma kernels when cutlass3x_sm120_bstensorprop_... CASK kernels are available, has mad Python in the hot path, emits dubious tool call syntax, and just managed to do something so off-road it tripped a driver bug that managed to wedge MMIO so bad the EC couldn't get an SBR out so the fan was going literally max until I hard pulled power. They say AMD is worse and I'll take their word for it because patching the driver and Inductor both in the same day is plenty of grief for me.
But the amazing thing and why your comment prompted me to write all this is that Qwen3.6 is insanely good, so good I am seriously questioning the MoE dogma. I was running the NVFP4 quant with the BF16 drafter, 100+ tokens a second of clean, legible reasoning trace and flawless tool calls. It's small and you can tell it hasn't memorized half the internet, but it's reasoning is like, better than most frontier. Opus has cleaner reasoning, GPT 5.x and Gemini 3.x Pro do not. If someone scaled that boi up by 5x? I get the feeling DeepSeek did so many arch innovations in one release that they just didn't quite have the convergence, this is like, the fundamentals as artistry. It's wayyyyyy stronger than GPT-4o at over a trillion parameters.
The other thing is I was using it on OpenRouter and it was all janky in the traces, stuttering and going in circles. On another day I would have been like "what do you expect it's the size of an iPhone". I wonder how many other people have drawn that conclusion too.
I'm not going to call it a conspiracy because it's explained by neglect, but we haven't even scratched the surface of the local model ceiling. With a harness that kept the data fresh, scale up the parameter count a bit, stretch context out a bit, and write an LLM serving engine that isn't hobbyist Python jank wrapped around fuckin inductor/triton jank?
That's Claude Code Opus experience on an expensive gaming box.