Comment by abhgh
4 days ago
I like this post. I haven't had time to dig into Jev (they aren't accepting new signups), but calibrated probabilities is one of their pitches that caught my attention. And I was wondering how does one offer them on user data. Standard calibration essentially ensures that if a score of 0.8 accompanies a positive prediction (assuming the simple case of binary classification), then if you gathered together all predictions with a score of 0.8, around 80% will be correct.
If you have just one example you're sending to a model, how would they guarantee 80% over your data?
FYI, for an overview, scikit's page on calibration is great [1], and my answer on Quora from a long time ago covers a specific type [2].
[1] https://scikit-learn.org/stable/modules/calibration.html
[2] https://www.quora.com/How-is-isotonic-regression-used-in-pra...
While I don't believe they are doing the following: you can calibrate by inspecting the reasoning traces. That is the relevant distribution. If you ask someone to explain how/why they are classifying something one way v another, you can get a reasonably good understanding of their confidence level.
This tells me the confidence of the LLM's belief about the response - which is different from the calibrated confidence score. The former also is useful (just not what I thought their advertisement sells - and from the article it seems like it tripped up others as well), and there are different techniques to extract such a value [1] [2], typically via "response sampling", i.e., interrogate the LLM slightly differently to see if it changes its answer.
[1] Semantic Entropy https://www.nature.com/articles/s41586-024-07421-0
[2] Kernel Language Entropy https://openreview.net/pdf?id=j2wCrWmgMX
I mean the model can learn from it during RL training. The confidence score is affected by the tokens prior to it it's output. I was using the word "you" loosely.
It's on OpenRouter if you want to try it.
Thank you!