Comment by satvikpendem
2 days ago
As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc.
[0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
2 days ago
As usual, the Jinja templates are messed up so use this [0] to reduce or turn off thinking, fix tool calling, keep a 100% KV cache hit rate, etc.
[0] https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs?
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
You have understood correctly.
One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.
But they don't.
Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.
I also don't understand how this repeatedly happens.
I did SFT / RL post-training on Qwen3 models a bit. This is an issue that dates back long ago.
My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.
Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.
Is the chat template used at all when they benchmark the model?
3 replies →
Yes, I was fixing issues piecemeal until I found the froggeric template, I've had to fix I think one issues with that one but it's better.
The chat templates are usually the first thing that every major release bork on, and all new model architectures end up having a ~2 week initial window of small fixes before they’re not DoA
Yes, and this is not the first time they messed up. They had tokenizer bugs where the trained weights do not match the template back to Qwen3 series.
Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
For those us us who don't know, what do those parameters do and why are they better?
10 replies →
The Qwen team published the same sampler settings for 3.8 and presumably they used those while testing on benchmark. Do you believe they could have achieved higher result with top-n-sigma?
How is terrible settings a case of FUD?
Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.
2 replies →
Interesting. Why don't the unsloth guides (https://unsloth.ai/docs/models/qwen3.8) mention this? Do they already include the fixes in their GGUFs?
They usually do include fixes yes.
@danielhanchen may be able to answer this
Thanks this bumped my agent success rate from 67% to 92.5% (!!!)
Depends on your tooling and quant? I grabbed the unsloth Q3 and it works out if the box in opencode. I had issues with OpenWebUI with a random 3.6 A3B.