Comment by the8472
7 hours ago
Yeah that's pretty much what gwern argues here[0]. Or to adapt another proverb: to predict the next token you first need to model the universe.
[0] https://gwern.net/scaling-hypothesis#gwern-difference--effic...
7 hours ago
Yeah that's pretty much what gwern argues here[0]. Or to adapt another proverb: to predict the next token you first need to model the universe.
[0] https://gwern.net/scaling-hypothesis#gwern-difference--effic...
> to predict the next token you first need to model the universe
Exactly. The "most likely next" series of tokens, for example, when given the first half of a correct mathematical proof, is the correct rest of the proof. I have never seen anyone define "most likely next token" in such a way that this isn't true.