Comment by dbbk
15 hours ago
When they say "can't hallucinate" they mean they produce a confidence value for every result, so you could see for example it has 0.1 confidence, and you can disregard the result - that'd be different from hallucinating where it believes it's correct
if it puts a high confidence value on a wrong answer, thats still hallucinating, no?
llm hallucinations are high probability tokens that are incorrect vs the real world
No, I don't believe so. Hallucinations are not "high probability" in a real sense. They are an artifact of the random walk the inference algorithm takes, which causes it to latch on to and chase attractors in the noise. This random walk behavior is necessary for chat interfaces to be useful, but are less critical to typed output predictors. I'm guessing they found some optimization that is possible if you give up caring about chat.
Correct, they have not made a universal all-knowing omniscient oracle, which is what would be required for "can't hallucinate".
That seems like a weird standard.
I would be happy enough with: only produces what it can verify with sources.
If you eg try to remember a court case (ie produce the reference via LLM token generation only), it's easy enough to check with your data whether it really exists. Similar for following links and other references.
If your data or sources are wrong, obviously your report about them will be wrong. But I wouldn't call that a hallucination.
4 replies →
Not to be tooo pedantic, but a bot that assigned 0 confidence to everything wouldn’t hallucinate.
A calculator either gets the right answer or doesn’t answer.
It wouldn’t have to be all knowing as long as it knew perfectly what it doesn’t know
2 replies →
What we would want to see if a confidence value that is in line with the actual correctness. If the value is 0.9 for 1000 different answers, then approximately 900 of those answers should be correct.
Yes, there is no magic sauce here that makes stochastic output binary if that’s what people are looking for.
Right and so maybe we should stop saying "can't hallucinate" when it can by definition.
It’s not what people are looking for, but what they wrongly claim.
Technically speaking when you send the prefix “The capital of France is “ into an LLM it will also produce probabilities across its whole vocabulary.
The probability values don’t really represent confidence in modern LLMs though, especially after RLHF and RLVR.
System One says they use RLCD, Reinforcement Learning for Calibrated Decisions, which presumably has accurate probabilities as an explicit optimisation goal.
How is that different from RLVR?
1 reply →
… which they could provide in their APIs but are vehemently opposed to because it makes distillation much easier, and faster.
Yeah. Yet another reason why open-weight models are better. If I want to use the logits, I can.
that's right, but because these models are probabilistic, it's also possible to be confidently wrong (and all future models will be smarter still and still have that possibility)
I read "hallucinations" as "generates novel output with no grounding/source". i.e. "it just made something completely up".
I believe their "accuracy" metric (sonnet 5 level) is where "right/wrong" is measured.
Yeah but what stops it from producing confidently incorrect outputs...
Nothing, but imagine using LLMs for a classification task
People out there are so resigned to the models being unreliable that they are really doing things like hallucinating deliberately, and then matching the hallucinations to embeddings -
https://softwaredoug.com/blog/2026/08/10/hypothetical-classi...
You could do that or you could just... use a model that will never produce unreliable outputs in the first place.
But we're going from "Apple" to "Apple: 99% - trust me". It could still be an image of an orange :)
2 replies →
I'm certainly not resigned to that, at least for classification.
Even non-frontier models are absurdly good at this in a broad sense.
Which would make it hard to judge "a model that will never produce unreliable outputs in the first place" against something that is already really, really good and exceptional in domain-specific areas with the tiniest amount of elbow grease.
Speed and cost look good though (for now)!
What about the LLM calls though that are done midchain? In the Home Assistant video the multi-intent prompt gets split using what looks like a traditional llm model, which I'm assuming is vulnerable to classical hallucinations.
That's really funny when you consider that generative models also don't hallucinate if you check up on them on every token generated?