Comment by TheOtherHobbes

It doesn't need qualia or consciousness, it needs goal-seeking behaviours - which are much easier to generate, either deliberately or by accident.

There's a fundamental confusion in AI discussions between goal-seeking, introspection, self-awareness, and intelligence.

Those are all completely different things. Systems can demonstrate any or all of them.

The problem here is that as soon as you get three conditions - independent self-replication, random variation, and an environment that selects for certain behaviours - you've created evolution.

Can these systems self-replicate? Not yet. But putting AI in everything makes the odds of accidental self-replication much higher. Once self-replication happens it's almost certain to spread, and to kick-start selection which will select for more robust self-replication.

And there you have your goal-seeking - in the environment as a whole.