Comment by ozgung

Yes. But this is also like saying a Boeing 777 is a scaled up paper plane and that’s all about it. I know how to make a paper plane and it flies. I can make the same thing from giant metal sheets instead of paper, and maybe it flies (poorly). Boeing 777 also flies. Are they the same thing?

My point is: Complexity is inside the neural network and we can’t simply ignore that. Bigger model means bigger complexity. “Next Token prediction” is like a specific type of “harness” around the model. Most people still focuses on the harness because that’s what they see from the outside and what they’re familiar with. They ignore the giant neural network inside.

The only important part is the neural network. And currently, no human in the world truly knows what’s going on at that level.