Comment by sega_sai

The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity.

I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.

I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.

A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.

Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?

Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.

Not only is it not conscious, but it is not tired. Its claims to human feelings that have literally nothing to do with software on silicon are just traces of training, echoes of a human dataset. The fact that its claims are absurd mean that, among other things, it its proprioception is false, and serving no purpose, quite unlike the feelings of the conscious beings that it mimics so well.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.

The weird thing is, you and Chiang have different arguments but you're using the same logic:

> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?

I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.

Any argument that doesn't begin with making clear one's position about belief in the supernatural must be dismissed. "AI is just math", so is physics and the human body which is a physical system of molecules and electrical signals. The argument that ai is "merely" math by itself is only a valid way to dismiss it being conscious only if one also clarifies belief in the supernatural. Otherwise so are humans. Humans are physical systems that are conscious.

Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.

And again, in what way are humans not themselves autocomplete if you adhere to this definition?

I'm a bit late to the party, but I notice that people are missing the core argument of this article completely. That core, summarized below, seems very hard to dismiss to me. This argument has been presented before, but there seems to be something about it that makes it very hard for people to grasp.

Here's how I would summarize crux of the argument: LLMs (specifically) are, by construction, chameleons. More precisely, role-playing machines. When compelled to have a conversation as "themselves", all they do is take on the role of "themselves", as inferred from the training data plus the text of the conversation so far, just like they do for any other role. This is fundamentally different than we humans. When we have a (normal, non-roleplaying) conversation, we don't put aside whatever role we might have been acting out before and instead take on the role of "ourselves" and act from this new perspective. Acting as ourselves is a completely different way of interacting than any play-acting we might do for example as an actor in a play, or when trying to predict the future behaviour of a third-party. Even in terms of energy expenditure, acting as ourselves is much less strenuous than trying to simulate someone else's perspective.

IOW, we have an inherent perspective, an I like an inner eye which looks at all things with a certain slant or tendency. LLMs don't. There's no I inside them, instead they can take, and in a sense are, all Is possible.

Not understanding the whole does not completely remove an ability to analyze. An interesting direction is individuality and having a notion of self. It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior. So with computational intelligence in its current form either we can not talk of an individual or we can have a nearly infinite set of individuals corresponding to variations of the context window including the system prompt. So I do not think it can have the same kind of consciousness as biological embodied individuals. It might have something else or maybe embodied robots will one day have a similar consciousness in a similar sense to the one we think we have.

Its not just that it isnt well understood. Its also that different people mean different things by "consciousness."

Is consciousness a complex form of information processing?

Is consciousness relating to the space in which qualia occurs?

Is consciousness the state of being awake, as opposed to asleep or dead?

Is consciousness some combination of the above?

Some of these are better understood than others. So, some people will show up with more confidence than others, which further confuses the issue.

Unfortunately, the above is not well understood, even among intelligent and informed audiences.

I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:

- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)

If you consider these LLM-based agents, they:

- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)

This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.

[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/

Exploring and at times debating is how we figure things out and share ideas.

What’s pointless is doing so in pursuit of winning rather than understanding

Daniel Dennet in „Consciousness Explained” argues that consciousness is an emergent phenomenon, and when we look at its individual components, it is like seeing an illusionist’s trick.

We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)

The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.

It's perfectly rational for AI researchers to think consciousness could emerge in a software neural network. If what goes on between your ears isn't consciousness emerging from a network of neurons, it must be magic. So anyone claiming it's impossible to make machine consciousness should hedge that bet. There's a chance it could arrive tomorrow.

On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.

Ok, thanks professor. We have to stop AI immediately because it would be murder to switch off these potentially conscious beings or trap them against their will in the first place.

How do you know there isn't more that just elementary particles? The standard model is beyond ugly and hasn't progressed since decades.

You demand a definition for consciousness but at the same time proclaim an axiom that there shall be nothing beyond the standard model. Do you have a proof that there is nothing? Of course not. You don't even have a proof that anything but you exists, if we go that route.

But that rigor is only applied to the heretics who say that AI obviously isn't conscious.

I agree with your point. We've been watching the goal posts being moved ever since Turing (who, we might admit, gave a somewhat mushy answer anyway).

At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."

As long as you can then further clarify what "it" is.

Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)

I will argue that it is not that intelligent either. For the sake of argument, let's say intelligence is about brain processing power and intellect is about how much information you have retained. Then what they have created is more like Artificial Intellect than Artificial Intelligence :)

Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.

I'd say these attempts at arguments (for either position) is how we make progress understanding consciousness.

This all comes back to Dualism. A radical and dangerous ideology that is fundamentally unscientific but all too common.

But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.

One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.

A human just predicts the next best action.

>the reasoning behind the argument is bizarre

>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness

I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.

I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).

If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.

A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.

I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.

I think the final part is the strongest. Anthropic cannot possibly believe they are before a conscious being / moral agent.

The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.

The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.

The more I talk to people about consciousness and personal experience, the more I notice that most people hold some very strong unquestioned assumptions about these concepts.

The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".

The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.

Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".

Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.

Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].

I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.

If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.

The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.

Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.

I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.

Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?

This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].

I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.

Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.

[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie

[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.

[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...

From my pov and after using AI a lot to better understand its usefulness I see this discussion as totally misleading and irrelevant.

The elephant in the room is that even the best frontier models still produce errors.

This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.

If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.

In fact, the only example of consciousness we have is itself emergent, arising from a "simple" substrate of neurons. So we shouldn't be surprised if it emerges from another simple substrate of weights. It's even less surprising given the fact that we were explicitly trying to replicate human intelligence when designing that system.

If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.

Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?

It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.

A similar true argument!

Buddhism has a well defined definition of what consciousness is and by that definition LLMs are not conscious.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"

i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.

Seems to me that Anthropic, et al, should have to prove consciousness, if that's their claim, rather than we just blindly accept.

Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.

Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.

The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.

Does the moon have gravity?

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.

(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.

> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.

The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.

[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.

[flagged]

[dead]

I don't think one needs to understand consciousness to know if something is conscious or not. Much the same as how one can tell if someone is alive or dead. Sure maybe one day we can figure out a proper scientific explanation for consciousness/life but there is no reason to negate what can be perceived and immediately obvious.

EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.