Comment by kmike84
14 hours ago
Measuring quality e2e definitely makes sense.
But I think there is a bit more to this:
> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.
A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.
I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.
You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.
Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.
Still investigating it though :)
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.
On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.
So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
Yeah, makes sense; e2e is different and valuable, KLD is not a replacement.
As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
Nope.
If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.
> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext
Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing:
Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baseline for stats)
Gemma 4 26B: "Dude, I'm certain I wouldn't have said this." (Bad KL)
Gemma 4 12B: "Umm, Me either!" (Similarly Bad KL)
Gemma 4 E4B: "I might say almost anything, this is fine." (Much better KL!!!)
Gemma 4 E2B: "I'm basically a toy. Let's play a game!" (Same KL as E4B)
Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.
> Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.
Way better than wikitext-- but tells you nothing about errors tending to compound or cancel out.
Like say a test shows that only one token in a 10,000 token test would be different. Sounds very close, ship it!-- but what if trajectories with that single different token guarantees failure because it sets in motion a cascade of differences that ultimately result in a final distribution that doesn't include the solution?