Comment by jrflo
4 hours ago
I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
I’ve run those experiments. Commercially, so I can’t share data. But I can say what the article describes roughly matches what I’ve seen. How well calibrated the scores are varies by model, application, and subtle wording changes. I’m a little more optimistic than the author in that I have gotten it to work reasonably well. But I would not recommend productizing it with a SaaS LLM because it will break every time your vendor sunsets the model you were using, and getting it working again on a new model will be an unholy PITA.
that's the annoying part about LLM providers, they don't tell you anything if the model changes behind the API. I hope open models catch up
You mean like this recent preprint?
https://arxiv.org/html/2607.20526v1
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
[dead]
I would guess it tends towards more likely because it can't not think of something mentioned.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
I think you're making the wrong takeaway here. LLMs aren't bad because of sycophancy tendencies. Any prediction at all is bad when it isn't based on real data, whether or not it ends up being correct, whether or not it is made by an LLM, a human, or some other thing capable of spitting out a number.
Analogize to basketball here. How confident are you that you can hit a free throw? If you ask someone who has played a thousand hours a year for the past ten years, has recorded their own statistics, and has averaged 70% success rate oer those past ten years, and they tell you they're 70% confident they can hit a free throw, that's a good prediction. If you ask me and I give you any prediction at all, I'm full of shit. I haven't played basketball or taken a free throw in 30 years. I can pull a number out of my ass, I can reason through it, I can do a Fermi, I can assume I'll be the average of all people who have ever tried and do my best to collect whatever data might exist on brand new players with no experience trying to take a free throw for the first time. I can narrow that to people my height, my age. Now we're getting better, but it's still not as good as having a real record of my actual past performance and basing a prediction off of that.
This is just science. Hierarchy of evidence. Epistemology. Sometimes answering a question requires collecting data that doesn't exist yet and waiting a while, in the meantime admitting you don't have an answer. It was already sorely lacking in the world, but in this age of oracles, it's only getting worse.