Comment by TheOtherHobbes

As usual with these debates intelligence is confused with agency, abstract self-awareness, and subjective human-like consciousness.

Systems can have goal-seeking agency with zero self-awareness. Very simple systems can do this surprisingly well. There's an entire branch of engineering called control theory which has mathematical tools for modelling and building these systems.

Systems can have abstract self-awareness through introspection loops, but no subjective human-like consciousness.

We don't know if a mechanical system can have a subjective human-like consciousness. We likely never will, because there is no objective measurement device for human-like consciousness.

We tend to assume humans have subjective awareness by default, because life would get very complicated it we didn't. But really all we see are behaviours.

Intelligence - which pretty much reduces to pattern recognition and abstraction, and effective modelling of future outcomes from past inputs - is orthogonal to all of these.