← Back to context

Comment by lostmsu

2 days ago

> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.

Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.

2 comments

lostmsu

Reply
nullc  2 days ago

It's a cheap to evaluate proxy for totally broken or not, which is a good start.

It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.