The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
>The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.
Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.
The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.
If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.
I'm not even an LLM researcher. But I believe you're correct; I just don't think the paper provides any meaningful evidence on the subject. From the structure of the model we can tell that dots can do more computation than no CoT and less than full CoT. What we don't know is how much of the performance gap between no CoT and full CoT would be recovered by an LLM using dots. The paper answers that question for a tiny transformer, but it's so different from an LLM that extrapolating to LLM scale is purely speculative. In other words it doesn't give us more information then what we can already determine from the structure of the model.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Specifically, they train a model from scratch. The model architecture is apparently based on Llama but the size is 34M parameters. Not 34B, 34M. This is a fraction of the size of GPT-2.
Luckily, they don’t use the model as a language model. It neither receives text, generates text, nor uses text to think. Instead the inputs are strings like “A01 B10 C73 D27”, and the only possible outputs are “True” and “False”. They are expecting the model to solve a specific math problem encoded by those numbers, and do nothing else. The chain of thought is also numbers, in the scenario that’s supposed to represent a real chain of thought (as opposed to the filler-token scenario and the no-CoT scenario). The numbers in question are manually trained into the model based on one possible algorithmic decomposition of the problem; the model does not learn to generate its own CoT.
Even with all those limitations, for their main problem (3SUM), they only show that filler tokens are better than no CoT at all. They don’t show how that compares to ‘real’ CoT, at least as far as I can see (admittedly I only skimmed). They do make this comparison for their easier problem (2SUM), but on that problem both filler token CoT and ‘real’ CoT are mostly saturated, so the results don’t mean much.
>The model they use in that paper is a toy model of an LLM that’s so different from an actual LLM that I doubt the results mean anything at all.
Sorry, but it sounds like you aren't cut out for transformer based LLM research if you can't interpret the results.
Each layer in a transformer has its own KV cache that feeds from the previous layers output. Those internal layer activations are not subject to training loss directly. The model can spit out a dot but still perform additional computations within those layers.
The dot effectively prevents information from flowing from the final layer into the output, but all the intermediate layer activations are still available for future tokens. The limitation here is that information cannot be passed from the last layer to the first layer.
If a long running calculation needs to go through all layers and then be refined further, CoT becomes mandatory because CoT tokens are the only way to pass information up to the first layer, but since they are not as information dense as the internal activations, replacing them with a dot does not degrade performance as much as one would expect.
I'm not even an LLM researcher. But I believe you're correct; I just don't think the paper provides any meaningful evidence on the subject. From the structure of the model we can tell that dots can do more computation than no CoT and less than full CoT. What we don't know is how much of the performance gap between no CoT and full CoT would be recovered by an LLM using dots. The paper answers that question for a tiny transformer, but it's so different from an LLM that extrapolating to LLM scale is purely speculative. In other words it doesn't give us more information then what we can already determine from the structure of the model.
For the record, the original claim was that "you can replace intermediate tokens with single character chains and still get the increased precision", which I read as saying you can get almost all of the benefit of CoT with dots, not just some unknown fraction of it. This claim is true for the paper, though probably only because the benchmark was saturated (and training difficulty was also higher, as they mention). But the claim is unlikely to be true for LLMs, if only because they presumably do sometimes need to perform "long running calculations", and they also have plenty of unsaturated benchmarks.
(The other part of the original claim, that CoT is "not a representation of a model’s logical path", is more complicated. It's clearly not guaranteed to match the model's logical path, as we have seen with other research into actual LLMs. But it does seem to be correlated with it, presumably more so for more complex / long-running tasks.)
Also 2504.09762: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!
https://arxiv.org/abs/2504.09762