Comment by pegasus

What? Where are you seeing such claims? I.e. that only humans can be conscious, that AI in general is not conscious by definition?? Seems to me you've put up an easy-to-dismiss strawman.

Here's how I would summarize crux of the argument: LLMs (specifically) are, by construction, chameleons. More precisely, role-playing machines. When compelled to have a conversation as "themselves", all they do is take on the role of "themselves", as inferred from the training data plus the text of the conversation so far, just like they do for any other role. This is fundamentally different than we humans. When we have a (normal, non-roleplaying) conversation, we don't put aside whatever role we might have been acting out before and instead take on the role of "ourselves" and act from this new perspective. Acting as ourselves is a completely different way of interacting than any play-acting we might do for example as an actor in a play, or when trying to predict the future behaviour of a third-party. Even in terms of energy expenditure, acting as ourselves is much less strenuous than trying to simulate someone else's perspective.

IOW, we have an inherent perspective, an I like an inner eye which looks at all things with a certain slant or tendency. LLMs don't. There's no I inside them, instead they can take, and in a sense are, all Is possible.