Comment by metalliqaz
13 hours ago
I'm not an expert on LLMs but it isn't clear to me why it is a problem to keep reasoning inside the layers of the model when the internal state of the model has never been readable text.
13 hours ago
I'm not an expert on LLMs but it isn't clear to me why it is a problem to keep reasoning inside the layers of the model when the internal state of the model has never been readable text.
It's a matter of degree not a black and while traceable/untraceable difference.
With a regular N-layer transformer you get a token every N-layers.
With a looped transformer there is no guarantee how often you get a token, unless you go out of your way to limit looping.
OpenAI's Jakub Pachocki says the "computational graph depth" (number of transformer layers passed though) for Astra is currently never more than 2x that of GPT-4, and does express concern that traceability will suffer if this is not controlled.
The number of steps between token outputs is increasing.
I don't actually think this is a problem, but it is a further step towards inscrutability.
Agreed. Anyone who thought CoT output was a reliable guide to the actual reasoning process taking place was fooling themselves from day 1.
A few hours spent playing with DeepSeek R1 should have been enough to dispel that illusion, watching it talk itself out of the right answer in its CoT (or talk itself into the wrong one) and still emit the right answer in its response to the user.