Comment by svnt
1 day ago
Look at any recent CoT output where the model is trying to infer from an underspecified prompt what the user wants or means.
It is generally the first thing they do — try to figure out what did you mean with this prompt. When they can’t infer your intent, good models ask follow-on questions to clarify.
I am wondering if this is a semantics issue as this is an established are of research, eg https://arxiv.org/pdf/2501.10871
Right, and then look at any number of research papers showing that CoT output has limited impact on the end result. We've trained these models to pretend to reason.
If it's only pretending to reason, then how is it that the CoT output improves performance on every single benchmark/test?
> Right, and then look at any number of research papers showing that CoT output has limited impact on the end result.
Which research papers? Do I have to find them?
> We've trained these models to pretend to reason.
I have no idea why that matters. Can you tell me what the difference is if it looks exactly the same and has the same result?
Examples:
https://arxiv.org/html/2506.02878v1
https://arxiv.org/pdf/2508.01191
Anthropic themselves: https://www.anthropic.com/research/reasoning-models-dont-say...
They were approaching this from an interpretability standpoint, but the more interesting finding in there is that models come up with an answer that fits their training and context provided. CoT is generated to fit the anticipated answer.
In these studies, there are examples of CoT that directly contradicts the response these models ultimately settle on.
This is not reasoning. This is pretense.
When they say "pretends to" here they're talking about something quantifiable, that the extra text it outputs for CoT barely feeds back into the decisionmaking at all. In other words it's about as useful as having the LLM make the decision and then "explain" how it got there; the extra output is confabulation.
Though I'm not sure how true that claim is...
1 reply →