Comment by danielmarkbruce
4 days ago
While I don't believe they are doing the following: you can calibrate by inspecting the reasoning traces. That is the relevant distribution. If you ask someone to explain how/why they are classifying something one way v another, you can get a reasonably good understanding of their confidence level.
This tells me the confidence of the LLM's belief about the response - which is different from the calibrated confidence score. The former also is useful (just not what I thought their advertisement sells - and from the article it seems like it tripped up others as well), and there are different techniques to extract such a value [1] [2], typically via "response sampling", i.e., interrogate the LLM slightly differently to see if it changes its answer.
[1] Semantic Entropy https://www.nature.com/articles/s41586-024-07421-0
[2] Kernel Language Entropy https://openreview.net/pdf?id=j2wCrWmgMX
I mean the model can learn from it during RL training. The confidence score is affected by the tokens prior to it it's output. I was using the word "you" loosely.