Comment by fwlr

I gestured at it with “embeddings” but to spell it out, I’m more or less claiming that the only sense in which LLMs “understand” a given thing is as big list of all the things it is related to (implemented in the form of a vector embedding). And all those things it is related to, each one of those things is just a big list of yet more things it is related to, and so on. I do not mean that it is eventually hitting a “base case” that contains semantic meaning and then transforming it according to the relationship path it took to get there; rather, I mean that it follows enough n-th order relationship links that the shape of the relationship to the future thing (i.e. what it is generating) is constrained enough to pick output tokens on the basis of their relationship to the current token alone.

As an analogy: if I give you a stream of numbers and you notice that the delta between number n and number n+1 is always 2, you now know enough about the relationships between the numbers in the stream to pick the next number without ever knowing what the numbers were.