Comment by jiggawatts

Stephen Wolfram had a great description of this effect in the early days (GPT 3.5 era):

Machine learning trains the network to do... anything that you reward it for. If you keep training, it keeps getting better.

Next word prediction can always keep getting better.

At first, simply "learning" spelling is what makes the predictions better because tokens are word chunks, not always whole words.

Then, the models "run out of steam" and can't get any better by learning more spelling rules, but the gradient descent forces them to get better... so they do... by learning the rules of grammar.

At this point the AIs can output correctly spelled and grammatically coherent sentences, but the sentences ramble on about nonsense topics.

So what happens next as the models run out of grammar rules is that they're forced to learn the rules "above grammar": logic, world knowledge, coherent story telling, etc.

At some point they learn to output pages and pages of fluid, coherent text, but... if they're not smart, if they don't think, and if they don't know what they're talking about, then they're still "suboptimal" and their forced gradient descent will make them close those gaps.

Eventually, the only way they can improve at "next token prediction" is by building up to human-like intelligence, including an inner monologue, theory of mind, and everything.

We can even read their "thoughts": https://transformer-circuits.pub/2026/workspace/index.html