Comment by indoorfish
18 hours ago
I expect it's actually not wasted and there's meaning behind what seems like nonsense to us in helping it achieve it's goal. Which is mildly chilling but not unexpected.
18 hours ago
I expect it's actually not wasted and there's meaning behind what seems like nonsense to us in helping it achieve it's goal. Which is mildly chilling but not unexpected.
I think this is a known phenomenon: even in non-reasoning models, adding useless/filler tokens before an answer improves task performance. The model is doing some computation during the filler. See: https://arxiv.org/html/2404.15758v1
It is. Processing tokens is the only time model has to do computation, and if you ask it a tough problem, there is some minimal amount of computation it needs to perform to process and solve it - pre-CoT in particular you could guarantee failure by forcing model to be concise, and thus giving it less computational budget than necessary to compute the answer.
(This is I think where people parroting out "stochastic parrot" are stuck even today - not realizing that "predicting next tokens" is hiding arbitrary computation underneath, with token stream acting as input and clock signal...)