Comment by killerstorm
7 hours ago
No, layer looping increases effective depth, but it still has to go through decode. So it's more like they increased number of layers from 100 to 200 without increasing number of parameters.
"Latent reasoning" is rather trivial - you can just replace unembed-embed step with a MLP. But labs don't do that largely because they want to read the output of unembed.
No comments yet
Contribute on Hacker News ↗