Comment by yunyu
15 hours ago
It's pretty clear reading from these comments that most HN members have a 2023-era impression of LLMs.
Modern chain-of-thought models with RL post training on verifiable tasks + realistic environments + rubrics are worlds apart from models trained on a simple next token prediction objective.
More money goes into the rubrics and RL environments than individual training runs themselves.
(Yes, at inference-time LLMs still output words one at a time, much like human speakers. But don't confuse the mechanism with the training objective.)
Even with heavy RL post training and rubrics, the model is still fundamentally bound by the next token prediction mechanism at inference. Rlhf and cot just affect the probability distribution of which tokens get predicted next. Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.
> and a single hallucinated token can still derail the entire chain of thought.
Incorrect. As the OP said, that is a very 2023 understanding of how LLMs work.
Grab a new model from OpenRouter. Have it work on a task. Change a few tokens and have it continue the completion.
> Have it work on a task.
With or without a harness?
Have you actually tried this yourself? Of course it can derail it. Try to reflect on your interactions with LLMs without all the constraints like web search, agentic scaffolding, etc.
The same way that a “yes” or a “no” input from you can change the response, cot tokens are fed back into the model as input and can derail it.
5 replies →
When you speak or type, you speak one word at a time. When you move, you actuate one muscle at a time.
Does this mean that a single incorrect word or twitch will completely derail the task you’re trying to performance? Or will you, like any other intelligent being, recognize it and compensate?
Analogies are good for conveying meanings not proving statements. How human muscles or brain works has no bearing on LLMs.
> Take away the heavy agentic scaffolding and external feedback loops, and a single hallucinated token can still derail the entire chain of thought.
With reasoning models, a derailed chain of thought can be rerailed.
> can be rerailed
What rerails it?
But but but....I was told it was a stochastic parrot! I liked that idea because it appealed to my vanity, and it described the gibberish produced by older models with bad prompting, and that was enough for me thank you.
/s