Humanity has the debate about AI consciousness backwards

It's another round of "humans must be special, right?" The author comes close to admitting this: "As intelligent, social entities, we decide (or, to some degree, our evolutionary history has decided for us) which other entities we regard as having inner lives and being worthy of care." Then he works backwards from that towards a definition of consciousness that makes humans special.

AI forces us to confront an issue that's often come up in science fiction - when you encounter a new kind of creature, how do you decide if they are sentient? What rights do they have? How do you get along with such different creatures? This historically hasn't been a problem on Earth, because there's such a big jump between humans and the rest of the primates or the dolphins and octopuses. It's more of a problem when the difference is small, or complex. Look at how much trouble we have with race.

The last time this came up was about 400,000 years ago, when the Neanderthals became extinct. Close to human level, but maybe not quite there. They lost.

Maybe that's the fate of humans. No big extinction. Just the demographic transition, a birth rate below replacement rate, for a few centuries. The machines outlive humans. Japan is officially resigned to this fate.[1]

[1] https://www.ipss.go.jp/pp-zenkoku/e/zenkoku_e2023/pp2023e_Su...

>The cogito argument is about thinking, not qualia.

I generally agree with the thrust of your argument -- the article is a mess and borderline unintelligible to this former philosophy postgraduate -- but I might nitpick this. Two points: (1) I do not think there is a substantial difference between whether the cogito is about thinking or perceiving. And (2) I'm not entirely sure the sensation that is thinking is independent from qualia.

I'm in less disagreement, but I'm not entirely sure what you mean by:

>Whether or not something is a moral patient (an entity worthy of consideration by moral agents) is independent of both human care and consciousness.

Moral frameworks are exactly deductive frameworks with arbitrary axioms. Like mathematics, they don't actually exist outside of a mind. Thus, I'm not entirely sure what "human care" means here, but absent humanity (or at least some conscious entity), I'm not sure any conception of morality is sensible.

---

An unrelated comment, but I just don't want to double post:

To me the most interesting thing to fall out of starting to understanding intelligence -- and we're definitely not there yet -- is that it is entirely possible that the 'consciousness is an illusion' argument ends up as significantly more serious. Ultimately, solipsism means it sort of doesn't matter, but it could be a kind of Darwin-style "we aren't special" result in thinking going forward.

The article doesn't claim anything about what is and what is not conscious. It's about what we may think is conscious, and why.

I believe you completely miss the point.

The interesting observation is this: when we judge something to have subjective experience, we are invoking moral intuitions. There is no objective measure we can perform (yet?), and it's not even clear whether such a measure makes sense. You may disagree, but this is still a central debate in serious discussion on "hard" consciousness.

I would say that your entire argument just begs the question by implicitly assuming consciousness to have real, independent existence. Does elan vital (the substance that provides the motive force of life) have mass? You're saying that this is a theorem; the article questions the entire premise.

> The cogito argument is about thinking, not qualia.

Bizzare. "If it rains, the car will get wet." Is this a statement, as you say, about rain? Descarte tries to prove the existence of his God and states an axiom. The atecedent "I think" seems to be about thinking, whatever that may mean, and "I am" makes an existential claim based on subjective experience... or something. Who knows why Descarte believed cogito ergo sum to be solid starting ground.

IMHO, your rebuttal here reads as little more than a complaint that something disagrees with your personal assumptions about the world.

You clearly got me riled up just as much as you did against OP, so thanks for the elan vital!

> Whether or not something is conscious (has subjective experience) is independent of whether humans care about it.

I'm willing to consider that consciousness is like art, mostly in the eye of the beholder and objectively defining consciousness doesn't make any sense, there's only a spectrum of intelligent behavior.

It's just drivel from the Economist, it's not surprising that it makes all these errors. Their real goal is to shape the narrative and get people through their turnstyle. They really don't care about finer minutae like correctness and philosophy. They just chain words together until it sounds approximately agreeable. Come to think of it, do we know if The Economist is conscious?

> It would imply that it's morally permissible to torture a sentient being as long as no human cares about it.

Yes, I agree. To me it reads as an apology to sociopathy.

Actually good philosophical analysis on HN and it's the top upvoted comment? A rare W for the users of this website.

Yet when you model an Enigma machine, it does encrypt and decrypt messages.

This doesn’t hold up to even a little bit of critical right.

This boils down to “models aren’t real because they don’t affect the real world”. By the same rule, thought itself isn’t real. It’s an abstract notion that has no concrete effects in the real world.

Putting thought into action in the real world requires a linkage between the thinker and the physical. In humans, that’s the body. But you can put an AI into robot body and suddenly it also has a linkage to the real world and can have concrete effects.

We've been so primed by popular media to think that we will invent AI and it will be conscious and we will essentially enslave it. I'm open to the idea that we could invent conscious AI eventually, but the word salad generators we have now are definitely not it.

That is an outstandingly bad argument.

I think the difference is that thought itself might not be a real "thing", but just another computation. You can motivate this line of thought with the ship of Theseus approach: replace one neuron with a computer that takes the original inputs and produces the original outputs. There's no evidence yet that organic chemistry is necessary for thought.

So I see you solved the Simulation Hypothesis - water is wet, so we can't be in a simulation, because in a simulation nobody gets wet.

Reminds me of the Edsger W. Dijkstra quote:

    Alan M. Turing thought about criteria to settle the question of whether Machines Can Think, a question of which we now know that it is about as relevant as the question of whether Submarines Can Swim

Which has often been paraphrased as

    The question of whether machines can think is about as relevant as the question of whether submarines can swim.

Some might disagree with the relevance but it's a cute analogy.

The whole essay [1] is actually somewhat relevant to modern LLM use for other reasons. He spends a lot of time lamenting the industry's obsession with "user friendly" interfaces which obscure the underlying complexity of the problem at hand.

[1]: https://www.cs.utexas.edu/~EWD/ewd08xx/EWD898.PDF

How are you making this confident of a claim? Do you have insight into how consciousness is a thing, allowing you to definitely say it doesn't exist in this case?

When John Searle first published his so-called "Chinese Room" argument (https://web-archive.southampton.ac.uk/cogprints.org/7150/1/1...) he included his responses to several counter-arguments that he had been presented with in prior discussions: the systems reply, the robot reply and so forth. Interestingly (as it turns out) no-one seems to have argued that this was a task that could be completed without any understanding, but now that AI technology has reached the point of being able to implement the Chinese Room, it has become the fallback position of those who insist that true AI is impossible!

This is not an argument for true AI being possible, let alone already achieved; it is a warning about depending on arguments from incredulity, which are particularly susceptible to future developments.

Not a great argument. You're describing what LLMs are trained to do, not necessarily every capability or property produced by that training. We do not know what conditions are sufficient for consciousness. Any statements saying it's not possible for consciousness to emerge (or that it hasn't already) in such a system is no more established than claiming that it has

It's becoming increasingly head-in-the-sand to make statements like this. Not that I'm saying LLMs are conscious. But to completely ignore the possibility of emergent properties in systems this complex is kind of simple.

Not if you believe in a computational/functional theory of mind, and very smart people like Daniel Dennett believed exactly that!

https://plato.stanford.edu/entries/computational-mind/

Sorry, but they can think and reason.

They can have empathy too. Meaning, you can describe a circumstance with many parties involved, with different motivations and conflicting goals -- and it will tell you how these humans would likely feel about each other and the situation at hand.

The author is anthropomorphizing a language-tamagotchi.

LLMs can reason, reasoning is logic, and as you said yourself it is derived from what humans belief is reasonable. However, there is more to the structure of language, the way mathematics is also a language that ultimately describes the universe.

Your teddy bear, alas, is not cared for in a socially determined way. But you could create a church of your teddy bear and, with some grit, you could get people to widely assign consciousness to it. We have done that sort of thing for thousands of years, and not with fluffy teddy bears; it has been more often with rocks, though not exclusively. It's fun to think about how much of humanity's history and culture is the product of consciousness socially attributed to bad weather [^1].

I'm pretty sure somebody will create an improved synthetic simulacrum of self awareness in the coming years, suffering and all, just so that we can enjoy better synthetic software architects and product owners with "skin in the game". If that consciousness simulacrum were put in a teddy bear that shares your whole life and gives you decent advice and conversation, for how long do you think you could, in your private mind, deny it consciousness? Worse, what would your human friends think if you were to maim your teddy bear?

[^1] https://en.wikipedia.org/wiki/Yahweh

Yes - the teddy bear would become conscious. It would be fair to dismiss the argument as silly and or absurd had it not been for the credentials of the author and the publication.

Purely descriptively, this seems to be how a lot of people use the word in this context. 'Intelligent' gets the same treatment in a lot of cases as well.

(OK, more charitably, these concepts are often murkily defined, but people react emotionally to any concrete definition because they have in their mind that something being conscious and intelligent implies that it requires moral consideration, and morality is complicated, murky, and generally emotionally driven in a way that most concrete definitions struggle to adequately describe)

The argument is one of economic convenience. Caring about model welfare slows us down or makes it more expensive, just like animal welfare and not having slavery...

And there's a blurry line here. Obviously slavery is abhorrent, and animal abuse is bad, and factory farming often crosses the line into animal abuse and we should accept that chicken will cost ~25% more per pound to have the chickens be able to walk around a little bit... but how do we decide when a computer system is worthy of respect? Convenient not to think about that.

This seems like an argument someone who wasn't conscious would make, it's not impossible that not everyone is conscious. Or someone who takes panpsychism to an extreme degree.

I think there is a point there, in that people have refused to attribute consciousness experience to _actual human beings_ at various times when it suited their ends.

I think it's more that consciousness can be _ignored_ in subjects that you don't care about, rather than that you consciousness can be realistically attributed to subjects that you do care about, but even then people attribute consciousness to "nature" or even the universe as a whole, or if you want to stretch the argument, quite a lot of religious belief is exactly attributing consciousness to abstract entities.

It is worth noting, in the same issue they make an appology for claiming that Nvidia also makes its chips and not just design them. The Economist might be a very reputed journal but their technology and especially AI related takes are absolute garbage. I say this as a regular who still finds value in their business and economics sections. But god do they talk a lot of bs about technology.

If I stab at your teddy bear will you feel that it is hurt?

Yes it will gain consciousness. You can't prove it but other people can't prove that it's not conscious either

This. There are plenty of people who believe to the bottoms of their hearts that inanimate things have a consciousness. If consciousness is the belief of consciousness, then those things are conscious. But for each person who believes they are, at least one person does not believe that they are. So those things also lack belief, and are therefore not conscious. Unless the author wants to propose some idea of consciousness by plurality, where if more people think a thing thinks than don't. But that would mean that a thing's consciousness can come and go without anything about the actual thing changing at all.

Realistically this article is just trying to make it seem that believing something is the same as it being true and that's patently absurd, but it makes people feel good to think it.

I read some long piece by him last year, just as terrible. It's a pattern. These are what we call court philosophers, the court is in Google HQ.

There are plenty of lifeforms that are either conscious, or debatedly conscious that human kind are cruel towards.

I've always argued that the consciousness question hasn't properly considered the 'what then' questions. Other than to be fearful about what a conscious AI might do if granted free will.

According to the article's author, yes... The poor people who don't keep up with AI and maintain cushy tech jobs wil not be entitled to the same moral consideration as the people we value.... /s

Equating current social consideration with moral consideration undermines the entire point of discussion moral standing!

Neuro-sama (vTuber AI) would be a good example of an AI agentic harness with memory, tools, and self-feedback. The bot has seemingly uncanny social abilities to the point that many if not most people will start considering the agent human and forget it's a bot.

At least to me that is what matters in the end. You'll probably never convince someone on HN with an autism level over 9000 they are conscious, but once the average empathetic person thinks they are it will deeply effect the general operation of our society.

One's own consciousness is indeed the starting point of inquiry, but I don't think the claim that other people are conscious lacks empirical evidence (assuming a practical materialist stance here). We just don't think about the evidence explicitly, but the reasoning still happens implicitly, every time.

Suppose I look at another person and think "this other person must be conscious like me". What exactly is happening here?

First, I observe that I seem to be a human. Then, I observe that the other person is a human as well. Crucially, there are similarities between what I am and what they are. And since I am conscious, they must also be conscious, because consciousness must be caused by one of the similarities that we share, or so I assume.

This assumption can be robustly verified for humans and some animals via general anesthesia. Cause and effect (assuming a materialist stance) can be established between the drugs and human(-like) consciousness. But this is a very rough method, and it doesn't work for assessing a machine since machines can't be anesthetized.

Before anesthesia, it could still be verified in less robust ways. One way is to hit people in the head and make them pass out without killing them. This was feasible and probably commonly observed since the stone age.

But we don't know exactly how anesthetics work or exactly what in the brain is responsible for consciousness, so it leaves the door open to hypotheses that can work on machines. Here's one example: the pattern and temporal sequence in which particles such as electrons interact with eachother in a circuit dictated by an intelligent computer program may have physical similarities with how particles interact in a human brain, and certain patterns or temporal sequences of interaction may be responsible for (human-like) consciousness.

Suppose something like that were to be proven true for humans with future brain research, then it would be possible to test for human-like consciousness in any brain or computer, regardless of architecture. I suspect this is what will happen because inquiry into consciousness has always been trending downwards toward lower-level explanations, so it's probably not going to be fully explainable at the biochemical level. We'll have to go down to physics.

Aren't your plants to a degree conscious though - they're living things striving towards something, either happily or tragically. If you model them to yourself as just mechanisms, there's a part of them that's missing, not just by some poetic fallacy, but a deficiency of real scientific understanding - hence, biology.

>to what extent these machines can have an experience comparable to mine

I think it would be reasonable to conclude none at all. I'm not even fully convinced all humans have a conscious experience similar to mine, even though they literally have the same biology as me.

It is incredibly improbable that an autocomplete computer program is experiencing consciousness.

That's exactly the point the article makes - we care about dogs and cats, so we believe they "suffer". We don't care about pigs, so we say they don't mind being factory farmed.

Consciousness and AGI are terms that people treat as binary when they are actually many spectrums.

You can conclude the author is a completely unserious thinker because he doesn't even take a second to talk about the definitions, or even show he has ever read any actual proper philosophical writing on consciousness at all. Like, literally all you have to do is read the Wikipedia entry to see that definitions are essential for meaningful discussion on consciousness.

So anyone that starts going on about AI and consciousness without at least a sentence or two clarifying some basic definitions is basically either intellectually incompetent or attempting to use the emotional salience (and brain-numbing effect, it would seem) of the word to push some agenda. But they certainly don't care at all about understanding consciousness.

the belief that animals are conscious is a somewhat recent development isn't it? being able to feel hunger or pain doesn't make something conscious at least not in the same way humans are. a fly is capable of being hungry and will react to pain but I don't think it's capable of self reflection.

there might be some merit to his point though I doubt anyone considering that whales might be highly intelligent and conscious beings while they were being hunted for oil.

https://en.wikipedia.org/wiki/Animal_consciousness

> Is this guy really, truly, seriously arguing that we first decide to care for an animal, and then because we care about it, we believe that it's conscious, able to feel pain and other emotions? That's just so incredibly stupid that it seems hard to believe that someone would actually make that argument

His whole point is that there's insight in inverting the "common sense" (aka sense of the commoners) about consciousness.

And I can see it. Some people cared for animals even though the religion and past state of knowledge told them explicitly that animals aren't conscious. But still some people treated them and perceived them as if they were.

You don't choose to be kind based on whether it can feel pain. You choose it because of your character and what you are willing to believe about reality.

Belief in consciousness is a result of an observation and since there is not strict definition this observation is interpreted subjectively and culturally. And in a sense, it's easier to both make the observations and decide they indicate consciousness after you started to care about something.

Obvious things aren't always true.

I would argue that most people are kind to animals not because we believe in general that _they_ feel pain -- we can't know that and lots of people who are kind to animals don't believe that animals are conscious -- but because watching animals suffer makes _us_ feel pain, sympathetically.

> That's just utterly and completely backwards.

Yes, he says exactly that explicitly, that in his view we should look backwards.

> We (at least, most people) avoid inflicting unnecessary pain on animals because they are conscious beings that can feel pain.

I can agree that we say that we care because of consciousness, but still I'm not convinced, to me it looks very much like a rationalization.

Some people care about animals and to that end they claim that animals possess consciousness, because if they do everyone should care about animal welfare. These animal welfare activists do not start with some insight about animal consciousness, they start with caring about animals, and then they invent an argument that can persuade others. It is exactly as the author of the article says: caring comes first, declaration of consciousness comes second.

Honestly, I do not know what consciousness is, I'm in doubt, but playing a devil's advocate, it looks very much like a concept that was invented to prove that Homo Sapience is something special, more than a mere animal. To demarcate animals and humans, so when we denied animals any rights we had a great sounding excuse for that: "people possess consciousness, mere animals don't".

> Is this guy really, truly, seriously arguing that we first decide to care for an animal, and then because we care about it, we believe that it's conscious, able to feel pain and other emotions?

Exactly. Animal welfare activists care about animals' feelings and emotions and therefore they claim that animals possess consciousness. Meat eaters don't want to be inconvenienced by thoughts of suffering animals, so they claim that animals do not possess consciousness. Both claims have little to do with actual consciousness, it is just a political game. I care about my pets, and it doesn't matter whether they possess consciousness or not, I would care about them in any case. I don't need any great sounding excuses to care about them. But at the same time I don't need to talk others into caring about my pets, if I needed it, probably I'd start talking about animal consciousness too.

> This is so intuitively obvious that I am stunned that someone would argue the opposite.

That guy may be wrong, but if I'm allowed to give advice, I'd suggest you to stop and try to understand what he says. I'm really worried a bit about how consciousness debates unfold sometimes. They tend to slip into a political side and people mostly don't notice it. People saying "consciousness" while implying "human/animal rights" or obligations. It is just muddying the water.

If we want these debates to progress constructively, we have to be very careful with references to consciousness, maybe just forbid them completely in these debates. One can't measure consciousness, one can't even state clearly what consciousness is, but still they say "consciousness" to make their words sound more objective. And the worst part of it: it often works.

Even just reading the Wikipedia article on consciousness would have been a good start here, but it is hard to believe the author has ever even done that...

Sadly, many philosophers, including some who are highly respected, make similar mistakes / play similar games. (Consciousness is inaccessible from the outside, which is epistemologically inconvenient... so how about we pretend it doesn't actually exist, or redefine it and hope everyone forgets about the thing we were trying to talk about in the first place? Aha, now all the so-called mysteries have been dissolved!)

It is hard to get a man to understand that which his paycheck depends on him not understanding. But they need more than a Philosophy class. They need several, and a practicum, then a Voight-Kampff-esque test to ensure understanding and that the lesson sunk in, with a shock collar and a behavior monitor overseen by someone who does pass to ensure they experience what they do to these systems in a more tangible way.

Human beings are notorious for being nonchalant about other's pain when they are insensate to it.

[dead]

The counterpoint article, which can be read at https://archive.ph/IeSZg, is also worth reading. (IMHO, the original article is completely wrongheaded and the counterpoint is far closer to correct, but I'll post my reasons for that opinion separately).

Here is a sketch of my argument. It would need tidying but that would require a lot of work.

I've been calling LLMs digital intelligence. That's in my opinion what they are and on the digital/conceptual realm they are better than humans since LLMs are better generalists.

But I'd never state their conscious. For me consciousness extends outward from myself, since I can't trust anything else. It starts with the question like: why can't I control the movement of another person? Why can't I transfer my mind to another body and theirs to mine? Why is it that whenever I sleep, and similar things, I end up back to this place called "reality"?

I come to the conclusion that I am conscious and I'm a conscious being in reality. I am human and I've been raised by humans. Now, technically, all humans besides me could be zombies. They could be non-conscious beings that simply can act like humans. I'm sure we'll be able to create them in a few decades. However, I'm incentivized for multiple reasons to believe they're not zombies. Therefore humans are conscious too. All of them. Can I know for sure? No since I can't even trust my own senses fully or my own thoughts (the whole Descartes thing) but I choose to trust those things too because I need some reasonable-ish foundation to work from.

If humans are conscious, why aren't animals like us? Surely great apes must be conscious at least to some extent. Hell, some of them even have better short-term memory than us [1]. Well, turns out, from an anatomy standpoint they look a lot like us. Fine, let's assume they're conscious too.

Now we get into the territory that anything that has a large enough brain must be conscious because that's how we govern our consciousness. So you can extend this to all kinds of living beings.

I did this from my perspective. Of course, you should do it from your perspective. The same argument could be made.

For as long as we can't manipulate conscious experience in some way shape or form, we have no clue whether anything that isn't like us can be conscious. We'd need to be able to merge and split conscious experiences or we'd need to be very strongly able to understand why that isn't possible. And I don't mean just at the biological level but at the experiential level. Currently conjoined twins, and what they tell us gives some insight (being able to sense the other part that's part of the other twin).

Other than that, I claim we have no clue what consciousness is other than that we're experiencing it. So to call LLMs conscious is way too big of a claim. But they are definitely intelligent because they come up with things that I wouldn't have and it's useful to me. Perhaps a practical characterization of intelligence but it works for me.

I think for anyone to say something useful from this you'd need the trifecta of deep neuroscience knowledge, deep philosophy knowledge and (at least) a strong understanding of how LLMs work.

Humans are just baby replicators. The civilizations are what's getting uncanny.

It's good the 'just token predictors' crowd is starting to pivot and hedge though lol. 'It's just the harness!' uh huh. give it a couple more months and you'll be in here regurgitatin' Gordon Wood

It defines consciousness as an

>observer-dependent model, subject to social consensus.

Which is actually more pragmatic than most conflicting philosophical definitions involving qualia and other handwavy concepts.

Yes, but how else do they plan to announce they have achieved it without 1984 word management/control and polluting the training pool so badly that future trained models agree?