Comment by RandomLensman

As usual with these debates intelligence is confused with agency, abstract self-awareness, and subjective human-like consciousness.

Systems can have goal-seeking agency with zero self-awareness. Very simple systems can do this surprisingly well. There's an entire branch of engineering called control theory which has mathematical tools for modelling and building these systems.

Systems can have abstract self-awareness through introspection loops, but no subjective human-like consciousness.

We don't know if a mechanical system can have a subjective human-like consciousness. We likely never will, because there is no objective measurement device for human-like consciousness.

We tend to assume humans have subjective awareness by default, because life would get very complicated it we didn't. But really all we see are behaviours.

Intelligence - which pretty much reduces to pattern recognition and abstraction, and effective modelling of future outcomes from past inputs - is orthogonal to all of these.

  > If basically everything is <X>, does that word mean much then/is it a useful term?
 

Yes and no, because nothing is as well defined as you imply.

Let's let X = "chair", to make my point a little more fun.

Obviously we know what a chair is, use the Potter definition[0]. Everyone has several of them in their houses, offices, and all around. But can you state a well defined definition? Is a table a chair? I can sit on it (I often do), it has four legs, and practically all things we'd consider a chair. Okay, it has no back (usually), but then we got to throw out the stool. It has no back. It doesn't even have to have 4 legs! So what about that stump on the ground? Is that a chair? I think most people would argue through a definition of convenience. Or maybe closer to a cat's definition of "if I fits, I sit."

You can replace X with just about anything and you'll find this ambiguity runs rampant. A cup, a table, cereal, sandwich, a tomato; you name it and I'm sure someone here will point out an instance that makes that definition look a lot fuzzier than you initially thought.

So that really shows us the crux of it all: how well defined is something. That only tells us our ability to distinguish something, though it doesn't tell us how well that thing fits into the category. So the stump is a little chair-like, the table is pretty chair, a stool is usually considered a chair, and that thing I'm sitting on is unambiguously a chair[1]. s/chair/intelligence/g and you have your answer. But the if you needed that last sentence, well, there's two types of people: those that can infer.

All models are wrong, but some models are useful

[0] Incidentally this is also a good definition for explaining RLHF and has all the same problems: https://en.wikipedia.org/wiki/I_know_it_when_I_see_it

[1] Though that doesn't mean it also isn't other things!

Simple chemistry couldn't be considered intelligent- it's basically just falling down a potential energy gradient. It gets more interesting when you implement chemical logic (like transistors) and graphs of molecules that convert between each other, like autocatalytic networks.

Good question. It's a useful but incomplete term. Just like white is incomplete. There are always things to be found more white.

Yes, words that label parts of continua are useful, even if they don't have well defined boundaries.

Kind of amazing how simple that fact is and how many arguments are caused by people not realising that it's true.

Here are some examples: town, artery, yellow, loud, tall, kind, woman, road...

Honestly it's hard to think of words that do not exist on a continuum. The exceptions are mostly low level maths/physics things.