Comment by __MatrixMan__

I'm not an expert, but my current mental model for this sort of thing is that the thoughts were already there, somewhere in the training data.

Some human was looking for something like this once. They didn't find it, but they wrote about the search precisely enough that the finding can happen during inferrence.

Maybe somebody will come along and school me, but for now it's a fun way to think about it: A million dead ends, each with a uniquely disappointed human, now with a chance at a second life in the hands of a different human they haven't met. If only the weights had encoded enough to introduce us, supposing they still live.