Comment by stared
5 hours ago
Before this experiment I tried to run KLD on various context lengths to see a quantization-dependent deterioration.
On wikitext2 there was no difference. I concluded these have no long-term dependency and I should use Linux kernel. Still, the same.
So yes, KLD depends on the dataset. Still, it does not measure what any e2e test does.
Yeah, makes sense; e2e is different and valuable, KLD is not a replacement.
As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
Nope.
If you would like to do so, it is easy (and orders of magnitude cheaper) than running benchmarks.