Comment by timfsu
7 hours ago
Fascinating article. I daily catch LLMs in “lies” like: “I found the root cause of the bug” or “this approach is twice as fast”. It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards, but it’s extremely annoying. It’s one thing to have a LLM make poor decisions, but it feels worse to have it “lie” to you in the process.
This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates.
Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.
Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.
I don't see how "it's just token prediction" is relevant to that at all.
Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.
People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.
I have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process.
Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.
If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
2 replies →
> Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file.
That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help.
I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice.
It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work.
I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
That’s a bit of a garbage in, garbage out moment.
Why is this being downvoted?
> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards
LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.
Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
That’s exactly why I don’t believe LLMs will cure cancer anytime soon, make terrible lawyers, shouldn’t be trusted for medical decisions, etc. software and maths are some really the niches where we have great, battle tested, reliable validation tools. That’s not the case for “softer” domains
But this is exactly what the AI labs should be doing ...
Love LLMs gonna keep using them. It feels like your suggested approach is expensive, in terms of tokens. I feel (second time I say this) that when I steer the process I get pretty good results vs my coworkers that let the LLMs run away. I do have data on our token usage, not much in terms of quality of the delivery.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
Thr thing is that both you and your agent should have a way to verify the solution.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
> expensive, in terms of tokens.
No amount of tokens can come close to my hourly rate.
Do you steer your agents by manually running every single test and linter and reporting the results back to them?
I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.
A lot of weirdness in their style is the fact they think in-band.
Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.
To write an honest answer, it says, "Here's the honest answer", and proceeds from there.
If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".
This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.
You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.
LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions.
In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.