Comment by skrebbel
2 days ago
I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs?
Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?
You have understood correctly.
One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.
But they don't.
Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.
I also don't understand how this repeatedly happens.
I did SFT / RL post-training on Qwen3 models a bit. This is an issue that dates back long ago.
My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.
Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.
Is the chat template used at all when they benchmark the model?
It’s unlikely they are benchmarking that downstream. They probably have private benchmark scripts with purpose-specific run telemetry and logging, etc.
I’m sure there is some basic testing but it might be agentic (LLM likes its own output) and maybe just some human smoke tests.
1 reply →
they likely use their internal infra to run benchmarks; aligning external releases with internal environments is always painful and somewhat underincentivized
Yes, I was fixing issues piecemeal until I found the froggeric template, I've had to fix I think one issues with that one but it's better.
The chat templates are usually the first thing that every major release bork on, and all new model architectures end up having a ~2 week initial window of small fixes before they’re not DoA
Yes, and this is not the first time they messed up. They had tokenizer bugs where the trained weights do not match the template back to Qwen3 series.
Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.
If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.
For those us us who don't know, what do those parameters do and why are they better?
Temperature, top-up, top-k, min-p all control which token the model predicts next and how likely it is to select one token over the other.
You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.
That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.
7 replies →
The Qwen team published the same sampler settings for 3.8 and presumably they used those while testing on benchmark. Do you believe they could have achieved higher result with top-n-sigma?
How is terrible settings a case of FUD?
Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.
Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...
1 reply →