Comment by famouswaffles

LLMs learn to make predictions. They don't learn to imitate. They don't learn to simulate. There's nothing about learning to predict that makes the intelligence you gain constrained to the data you're learning from. The opposite if anything. But that's another argument for another time.

The point I was making is that LLMs don't come out of training the average of what they've learned. They can make predictions on any state in their training.

They can make predictions about the most intelligent state and the dumbest state. The most emotional state and the least emotional state. It's this powerful prediction range that makes them capable of imitating or simulating damn near anything in the training set to high and ever increasing accuracy.