Comment by verdverm
7 days ago
Preliminary analysis via lm-evaluation-harness + vllm
model | disk | wikitext | gsm8k (match/error)
baseline | 55G | 8.00 | 0.50/0.09
nvfp4-gptq | 27G | 8.25 | 0.47/0.9
nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9
bonsai-4bit | 19G | 16.75 | 0/0 (eval bug?)
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope)
Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
Doesn’t this suggest you aren’t properly running the model?
Its unclear if its in vllm, the eval harness, or the weights. I'm running it the same as other qwen3.6 derived models and it appears to work, but other comments speak of waiting for support in their tool. Could be a chat template or could be legit because I'm using a different quant / bit selection from their hugging face and using their primary model as the comparison point for my huh? They may have put more effort into the flagship model than the 4bit because they are focused on on-device model running.
Software was already complex, now we are adding highly non deterministic elements into the mix