Comment by kennyadam

>without an actual understanding of what they're doing.

At what point do you start to question your assumptions that are causing you so much cognitive dissonance?

But to answer your question: to predict the next token really well you just have to model the world. Think of it like this, a simple statistical model might say "when token A is seen respond with token B". The next step will add conditions, "...respond with token B unless X has been seen, then respond with Y". Add a few billion more of these contexual clauses and you have a sequence of logical rules that indirectly model the relevant processes in the world.

“Next-token prediction” describes the output format, not the computation required to choose each token. During training, models develop internal representations of concepts, constraints, possible futures, and algorithms.

The PCB agent also writes circuit code, runs simulations, reads failures, and revises the design. It isn’t one-shot autocomplete.

Astra and Fable are already hard to square with “mere autocomplete.” We may be (really) close to AGI, and token-by-token generation certainly doesn’t rule out subjective experience (I think we should at least treat that as an open question).

Great videos: https://www.youtube.com/watch?v=D8GOeCFFby4

https://www.youtube.com/watch?v=Bj9BD2D3DzA

https://www.youtube.com/watch?v=l6DKRf-fAAM

https://www.youtube.com/watch?v=GlYgs6v2YfU

Look up "mechanistic interpretability" in the context of LLMs. The next token prediction machinery is just a foundation for a higher order learned structure that appears to encode specific concepts, regardless of input language.

The analogy to humans is that the human brain is "just atoms bouncing around", but there's unquestionably something "more" going on that just that.

My 2¢:

When google trained a neural net on Go moves, using some text notation for them, with no other vocabulary of any kind, just predict the next go move, they noticed a representation of a Go board had essentially formed in the network, all on its own. It had never “seen” a go board, or had one explained, but they could map neuron states to go board squares pretty much 1:1.

I truly think that LLM’s with hundreds of billions of parameters in their neural networks have all kinds of hidden “models” of things that arise from the simple act of predicting tokens. We’ve seen that the hidden layers in their networks model all sorts of program execution state for instance, when they’re working on coding tasks.

“Predict the next token” is a way of shaping/reshaping the neural network until it actually develops models of the things you’re giving it. Like the go board example. And I would wager that it has a compounding effect: once you have some useful models in the network, they can unlock the creation of other models, and so on.

"my brain hurts trying to understand how that can be"

Well, we all are, some are just more used to it by now and take the magic for granted.

My simple explanation, those neural networks save lot's of patterns of data, and that pattern can represent an image, a code snippet, a poem, or well ... description of a circuit board. And especially the text variant, LLM's - did copy all from us - so obviously they sound like humans, when they internally debate how to do something as this is what is in their trainings data how humans sound, when doing similar tasks.

But really understanding it? Not sure if there is a single person on earth who does.

> Yet, it's all the most likely token and my brain hurts trying to understand how that can be.

You and everyone else. That's the great mystery of transformer architectures as applied to language.

To be clear though, they're only good at schematic capture, which is very much a textual representation. Most of the data basically boils down to netlists, which are a text based format mapping connections between abstract pins that only later map to physical copper. The actual schematic portion is for human consumption and LLMs don't need to produce those to be useful.

Where LLMs completely break down is the next step, PCB routing. That's an NP-complete research problem that's been ongoing for decades without much progress. I've had some fun playing with using LLMs to better specify DRC rules in Altium so that the "classical" algorithms are more usable, but at the end of the day their geometric intuition is nonexistent.

type “shai next-token” and then “transformers learn shortcuts to automata” into arxiv and prepare to be blown away

Now go read Blindsight and enjoy the mental crisis.

humans when a machine better than them at spotting patterns appears:

You fell for the stochastic parrot meme and next token over simplification. That's the explanation.