Comment by TeMPOraL

Thanks for clarifying; indeed, I think we actually have somewhat similar perspective on this.

> I guess I am saying is that AI is much more like Alien than Artificial, but we read the tokens as if they are deeply human, and it is really hard for people to not think of it as human, purely because it uses language in such profound way.

That's something I keep thinking about, but I'm still of two minds about it. On the one hand, there's no reason to assume that a machine intelligence is going to be much like ours. You put it nicely:

> it can do theory of mind to us, but we can not do theory of mind to it.

And on the one hand (still the same hand), we shouldn't expect to. The design space of possible minds is large; if we draw one at random, it's highly unlikely to be very much like our own minds.

On the other hand, LLMs were not drawn at random. They're a result of brute-forcing a goal function that's basically defined as, "produce output that makes sense to humans", in fully general sense. And then, the input is not random - this is a point I tried to communicate earlier. You say:

> What I argue is that language is uniquely human, and it is how it is because of the human condition.

I agree, but then I also argue that language itself implicitly encodes a lot of information on the human condition. It's encoded in what we say, and what we don't say. It's hidden in pattern of responses, the choice of words, the associations between words, and how they differ across languages people speak. It's encoded in the knowledge we communicate, and how we communicate it.

I also believe that, at the scale of current training datasets, and with amount of compute currently applied, the models can pick up and internalize those subtle patterns, even though we ourselves can't describe it; I believe the optimization pressure incentivizes it. And because of that, I think it's likely that the model really is becoming an Artificial, lossy approximation of our minds, and not merely a random Alien thing that's good enough to fool us into seeing it as human.

Whether or not my belief turns out to be correct, I do have a related and stronger belief: that language carries enough of an imprint of "human condition" to allow LLMs to actually process meaning. The tokens may be coming to us from an alien mind, but the meaning as we understand it is there.