Comment by khalic
9 hours ago
I was under the impression that intermediate tokens (“chain of thought”) are _not_ a representation of a model’s logical path, with one study observing that you can replace intermediate tokens with single character chains and still get the increased precision…
CoT is both correlated and causal of the model's real computations, it's just imperfect. If you manually add "Let's wrap it up" in the CoT during generation, most LLMs will actually wrap it up (this is a commonly used trick in local LLM circles to get long-winded LLMs to stop reasoning). This wouldn't work if CoT text didn't affect the actual internal model logic.
How can they do that? Does state propagate between each token output?
That is, I were under the impression LLMs were just f(context), so chain of thought was f(...f(f(f(initial)+initial)+f(initial)+initial), i.e. y_n+1 = f(y_n+f(y_n-1)), where y_n is the nth output and f() the transformer inference function. Do they carry state across?
I mean at every layer in a transformer, the attention mechanism does a massive state transfer between tokens.
In a recurrent transformer, instead of projecting from the latent space to token space after a fixed depth, you take the latent embeddings and then run them again through the transformer. This causes more time to think because there's more mixing. You can run that as many times as you want for more thinking before projecting the embeddings back to word space.
Personally I believe this is similar to how humans think. The brain is a fixed size yet if we think longer we seem to be able to do more than if we just react instantaneously. This is because the brain feeds the results of our musings back into itself for further thought.
Yes dots increases precision, but not nearly the same increase in precision as having actual useful reasoning in the CoT
How can you possibly believe this? If this were true, why do open weight models think in real English tokens and not arbitrary characters?
The tokens inside the transformer are only projected into token space to train them. In reality they ought to be treated as their own thing. What's really gone on is you've trained the final projection to be sensible rather than trained the llm to think using words. This seems to escape a lot of people. You can throw random empty tokens into transformers to get them to think more. More tokens means more attention mixing which means a larger dimensional space in which to think essentially.
Link to this study?
[2404.15758] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models
https://arxiv.org/abs/2404.15758
The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
1 reply →
Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
https://arxiv.org/abs/2504.09762