Artificial intelligence is not conscious – Ted Chiang

The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity.

I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.

I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.

A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

I think that by debating what consciousness is not is one of the best ways we can gain a deeper understanding of consciousness itself.

The weird thing is, you and Chiang have different arguments but you're using the same logic:

> Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

IOW if we don't know what consciousness is, how can we call LLMs conscious when we haven't seen even the barest intermediary steps towards our nascent concept of consciousness first?

I think it's telling to your point that when Chiang describes what would make him think an LLM was conscious he starts using words like "believe" and "want" right away, because yeah as you say, we have no qualifications for what consciousness is.

Any argument that doesn't begin with making clear one's position about belief in the supernatural must be dismissed. "AI is just math", so is physics and the human body which is a physical system of molecules and electrical signals. The argument that ai is "merely" math by itself is only a valid way to dismiss it being conscious only if one also clarifies belief in the supernatural. Otherwise so are humans. Humans are physical systems that are conscious.

Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.

And again, in what way are humans not themselves autocomplete if you adhere to this definition?

Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?

Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.

Not only is it not conscious, but it is not tired. Its claims to human feelings that have literally nothing to do with software on silicon are just traces of training, echoes of a human dataset. The fact that its claims are absurd mean that, among other things, it its proprioception is false, and serving no purpose, quite unlike the feelings of the conscious beings that it mimics so well.

Its not just that it isnt well understood. Its also that different people mean different things by "consciousness."

Is consciousness a complex form of information processing?

Is consciousness relating to the space in which qualia occurs?

Is consciousness the state of being awake, as opposed to asleep or dead?

Is consciousness some combination of the above?

Some of these are better understood than others. So, some people will show up with more confidence than others, which further confuses the issue.

Unfortunately, the above is not well understood, even among intelligent and informed audiences.

Not understanding the whole does not completely remove an ability to analyze. An interesting direction is individuality and having a notion of self. It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior. So with computational intelligence in its current form either we can not talk of an individual or we can have a nearly infinite set of individuals corresponding to variations of the context window including the system prompt. So I do not think it can have the same kind of consciousness as biological embodied individuals. It might have something else or maybe embodied robots will one day have a similar consciousness in a similar sense to the one we think we have.

Ok, thanks professor. We have to stop AI immediately because it would be murder to switch off these potentially conscious beings or trap them against their will in the first place.

How do you know there isn't more that just elementary particles? The standard model is beyond ugly and hasn't progressed since decades.

You demand a definition for consciousness but at the same time proclaim an axiom that there shall be nothing beyond the standard model. Do you have a proof that there is nothing? Of course not. You don't even have a proof that anything but you exists, if we go that route.

But that rigor is only applied to the heretics who say that AI obviously isn't conscious.

I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:

- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)

If you consider these LLM-based agents, they:

- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)

This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.

[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/

It's perfectly rational for AI researchers to think consciousness could emerge in a software neural network. If what goes on between your ears isn't consciousness emerging from a network of neurons, it must be magic. So anyone claiming it's impossible to make machine consciousness should hedge that bet. There's a chance it could arrive tomorrow.

On the other hand we've got no idea whether that chance is so close to zero as to be negligible, or whether it's imminent. Just the fact we don't know indicates it's closer to zero. Not having a theory of consciousness is kind of a big architectural risk. It's like not having a theory of accounting when you're making an ERP product. Sure, consciousness in bio neural networks wasn't designed. But it took a few few product revisions to get there.

Exploring and at times debating is how we figure things out and share ideas.

What’s pointless is doing so in pursuit of winning rather than understanding

Daniel Dennet in „Consciousness Explained” argues that consciousness is an emergent phenomenon, and when we look at its individual components, it is like seeing an illusionist’s trick.

We may wonder how many grains of sand make a dune, of how many molecules of water make a liquid. (John Conway would argue that it takes a single spin-1 particle to have free will, but I digress.)

The same way, even if individual chemical reactions are simple (you don’t want to use that phrasing when talking with a biologists) or neural activities are simple (likewise, with a neuroscientist), it does not mean that the collective process is simple.

I agree with your point. We've been watching the goal posts being moved ever since Turing (who, we might admit, gave a somewhat mushy answer anyway).

At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."

As long as you can then further clarify what "it" is.

Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)

I'd say these attempts at arguments (for either position) is how we make progress understanding consciousness.

I will argue that it is not that intelligent either. For the sake of argument, let's say intelligence is about brain processing power and intellect is about how much information you have retained. Then what they have created is more like Artificial Intellect than Artificial Intelligence :)

Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.

This all comes back to Dualism. A radical and dangerous ideology that is fundamentally unscientific but all too common.

A human just predicts the next best action.

But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.

One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.

I think the final part is the strongest. Anthropic cannot possibly believe they are before a conscious being / moral agent.

The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.

The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.

>the reasoning behind the argument is bizarre

>Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness

I agree. I think the whole point of natural "intelligence" is to predict future events in order to properly plan actions for survival. The only difference is that next-word prediction happens in a different space (in a non-physical, textual form). But I don't think the distinction matters that much, because by the time the signals reach our brain's "intelligence core," they're already preprocessed multiple times. We can't see real physical reality, we "hallucinate" colors, touch, etc. (I think schizophrenia occurs when this kind of "controlled hallucination" goes wonky, but I may be wrong). So we're not that different from LLMs here.

I'd define consciousness as meta-intelligence, i.e. the ability to reflect on why/how a prediction was made (and make corrections to the pipeline). LLMs so far cannot properly explain why a certain prediction took place, but I'm not sure humans can fully either! I remember there was research showing that by scanning the brain (or some other signals), you can predict a person's choice before they're even aware of it. It's possible that our explanations are post hoc as well, and that the meta-cognitive ability to explain our own reasoning is as rudimentary as LLMs' (see: all the biases).

If you think about it this way, the only difference left is that everything we do is based on survival. LLMs don't have this goal. But I'm not sure it's relevant to the concept of consciousness.

A few months ago, I recreated Qwen3's architecture in 30 lines of code, and it gave me a sort of existential crisis: does it really take just 30 lines of code and a float array to recreate something that thinks and sounds almost like me? Is that all there is to it? There's often the argument that the brain is much more complex than 30 lines of code, which is true, but in my opinion, a lot of brain structures are basically archaic legacy systems (or auxiliary subsystems) that are not strictly necessary for human intelligence (which is bolted onto those legacy systems). If you carefully remove 95% of the brain (if you know where to remove), you can probably still have consciousness left in some form, it just wouldn't be very capable of surviving on its own.

I think our ability to debate consciousness in machine learning systems is greatly hindered by our subconscious existential fear: the fear that we ourselves can be reduced to non-conscious bits and simple mechanisms. In a way, it feels like a destruction of the ego.

The more I talk to people about consciousness and personal experience, the more I notice that most people hold some very strong unquestioned assumptions about these concepts.

The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".

The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.

Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".

Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.

Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].

I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.

If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.

The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.

Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.

I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.

Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?

This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].

I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.

Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.

[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie

[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.

[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...

From my pov and after using AI a lot to better understand its usefulness I see this discussion as totally misleading and irrelevant.

The elephant in the room is that even the best frontier models still produce errors.

This is a fundamental issue and should be discussed instead of some esoteric pseudo-scientific nonsense.

If the current systems can not produce reliable results they are useless. The promise of actual usefulness needs to be fulfilled in a very near future.

In fact, the only example of consciousness we have is itself emergent, arising from a "simple" substrate of neurons. So we shouldn't be surprised if it emerges from another simple substrate of weights. It's even less surprising given the fact that we were explicitly trying to replicate human intelligence when designing that system.

If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.

Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?

It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.

A similar true argument!

Buddhism has a well defined definition of what consciousness is and by that definition LLMs are not conscious.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

I don't think that this actually follows. Compare: "When Angels are not well defined in the first place, it is pretty pointless to debate if a pebble is or isn't an Angel"

i.e. even if X is poorly defined, you can still often say that Y isn't plausibly X.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

But we know what consciousness is, even if you don't know how to explain it in materialistic[0] terms. Intentionality is what makes consciousness consciousness. We know that LLMs - and computers in general - do not possess intentionality, not even sensory intentionality. There is no "aboutness" and no semantics to the content within a machine, never mind that the processes within LLMs are not reasoning or inferential processes. It is a sophisticated behavioral simulation that may be syntactically defined, nothing more. It is purely a matter of transeunt causes and not the immanent causality needed for something like consciousness.

(There's also the more general problem that we cannot say that physical computing machines are objectively computing. There is nothing about the physical processes - which completely define what a computing device is doing - that could be identified with computation in any objective sense. Rather, we human observers assign a computational interpretation to the machine's operations, just as we assign them to the ink blobs in a book or the liquid crystals you are reading on your screen now. Computation is something we do, and we have built machines to simulate it. We formalized computation - which is a process of desemantification and syntactic codification - and build machines guided by these formalisms. Formalization is what makes mechanization possible.

> Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of electrons by tiny distance would produce consciousness.

The movement of electrons as such cannot produce consciousness. The reductionist paints himself into a corner by strapping reality into a Procrustean mechanistic and materialistic metaphysics that redefines reality according to some dedicated schema, and then strains endlessly to find how some phenomenon can arise or emerge from the paltry remains. (Either that, or he embraces insanity completely and becomes an eliminativist.) It is magical thinking. I suspect this fallacious line of thinking is what causes people to believe piling more syntactic operations will somehow magically fuse into semantics and intentionality.

[0] "Materialism" here is the metaphysical theory. I am not claiming that consciousness - at least not all consciousness - cannot be a physical phenomenon. I am only claiming that the materialistic view of matter cannot account for consciousness.

Seems to me that Anthropic, et al, should have to prove consciousness, if that's their claim, rather than we just blindly accept.

Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.

Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.

The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.

Does the moon have gravity?

[flagged]

[dead]

I don't think one needs to understand consciousness to know if something is conscious or not. Much the same as how one can tell if someone is alive or dead. Sure maybe one day we can figure out a proper scientific explanation for consciousness/life but there is no reason to negate what can be perceived and immediately obvious.

EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.

I have a similar long term interest in this field.

It has been quite frustrating encountering arguments that have been extensively debated for years be presented as if they were new revelations.

In all my debates with people in the last few years I have primarily taken the position of trying to explain the problems with claims of certainty, and that lack of certainty permits possibility of the opposite. We should act responsibly around what might be possible.

There is also the narrative of "being too obsessed if you could to consider if you should" or similar claims of an unconsidered path forward.

Isaac Asimov wrote the first of the Robot stories in 1940, they were not written in isolation, it came from an awareness of the situation and the questions that must be asked. There was a community considering these issues. Asimov gave the wider public a view of some of those issues.

If we have a hundred years of people going "This is coming, we had better decide what we want it to be" and nobody listens to them, or frequently outright ridicules the need for considering their ideas, why is it now we are placing the blame on those who are now showing some success at what they told us they were attempting all along.

> I won't rehearse their ideas here

Is there an accessible way for a layman such as myself to read about some of these ideas (Really I mean philosophical discussion in general) without having to read entire books? Is there an active HN-equivalent or wiki or something?

My background is in computer science and linguistics. I wholeheartedly agree with you: We humans are dubiously-equipped to determine whether or not AI could be conscious.

I'm also super curious to learn more about the philosophers you referenced and their thoughts on this subject. Would you be willing to share some of your favorite examples?

Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):

> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.

I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.

So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".

For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary.

If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.

I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.

Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.

> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.

Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.

> of the form of original data streaming in and out.

Except this is not consciousness.

"If a machine has to learn to understand humans to complete text, then that is what it has to do."

But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.

>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...

This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.

The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.

A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.

That makes the claim that an LLM could not do a task the same as saying no function can do that task.

Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,

If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.

I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))

Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,

>If a machine has to learn to understand humans to complete text, then that is what it has to do.

A language model completes text based on the overlapping patterns of the training data.

There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.

Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.

Yeah. There are good arguments against LLM consciousness. This is not one of them.

I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.

Come on, I invented parts of this technology at Google and am baffled why this is debated.

We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.

Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.

His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.

Probably not, we're nowhere near the complexity of the human brain yet, there are also quantization limits to the human brain (i.e. molecules, quantum physics, etc) so to characterize them as having infinite detail is probably a bad modal.

If I'm going to be honest most of the people who advocate this type of thing tend to be, shall we say, crypto-duelists who really believe in a soul but not like intellectually but intuitively and keep trying to come up with excuse with it's not just meat. So like you can find philosophers advocating stuff like this but they tend to have a bit of an agenda.

I'm very open to the idea that consciousness is substrate independent. I have a hard time seeing why molecules could produce consciousness from an electro-chemical path, but not from a purely electrical path. Having said that, it should be very clear that LLMs are not conscious.

LLMs process language. I'd even go so far as to say that LLMs "think" and "understand", or at least, they produce a facsimile of thinking and understanding such that it's useful for us to reason about LLMs as if they think and understand. We're not used to interacting with a non-human entity with the capability to process language, so it's easy to ascribe human traits to these things. But their "minds" (insofar as they have anything like a mind) are completely different from ours. These things have language without consciousness.

Chimpanzees are conscious. Dogs are conscious. Maybe ravens and cephalopods? Who knows. These animals do have minds much like ours. Higher order animals are conscious even if they don't have language.

If we want to be really pedantic, a computer does not necessarily need to be digital. You could build a consciousness-coprocessor with analog circuitry that brings in what you say could be needed.

But honestly I doubt that there is a real requirement for that, surely you could just increase the resolution you're running the simulation at until the difference decreases sufficiently. Imagine using 256 bit floats for example.

I wish people would knock this off. There is zero path to AGI at the moment — and all the Anthropic/pentagon Sam Altman/AI-skynet stuff is scaring people into being fearful (and outright ignorant) to the actual uses of this probabilistic NPL tool.

You can't use AI atm without a human with proper knowledge spot checking and directing it. We should market it that way.

Consciousness is substrate independent, but that's not the differentiator here:

My toddler son is conscious, he feels happiness and sadness and have urges and impulses without needing to know the entire history worth of writing from human civilization.

A dictionary is not, and a LLM sized dictionary with optimized query is still not conscious.

It doesn't have anything to do whether consciousness is substrate independent, or even analog vs discrete, the overall implementation just aren't parallel.

There are quantized processes in the brain and there is also analogic computing. So either way is just a matter of time science gets there.

Wouldn't a conscious entity be able to grow its intelligence over time independently?

LLM's require offline training and dont actually learn from their "lived / sessions / chats ect". Those can be used for training data but its not like its an implicit part of the technology.

For this reason I would say LLM's are not conscious, or automatically conscious.

Each and every one of many billion transistors in a GPU works due to certain quantum effects on P-N junction. That does not mean GPU computation is nondeterministic, unknowable or magic.

It is perhaps very relevant what Chris Olah (Anthropic co-founder) said to pope..

I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.

https://www.anthropic.com/news/chris-olah-pope-leo-encyclica...

How certain are we about the theory our minds waveforms are continuous? Can we prove physical continuity, or just up to planck-length resolution?

I'm poorly educated on this, these are sincere questions. They are not intended as rhetorical regarding the point of the prior post.

I think the representation in a computer, the fact that it is merely stored instructions and data, destroys everything but domains of simulation and emulation.

A medicine for those who anthropomorphize LLMs is to run the LLMs deterministically (without randomness and memory files).

It feels very unnatural to get the same conversation verbatim at a different point in time.

There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.

You might be interested in Erik Hoel's more formal version of this argument: https://www.theintrinsicperspective.com/p/proving-literally-...

> It's not changed by the experience

The entire file is not changed, but the KV cache is.

> It doesn't remember anything

The model definitely remembers previous exchanges within the same conversation.

Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.

One can fine-tune the models and we do get a new version of Opus every few months.

But you could argue the brain is just a bunch of coordinates describing spatial relationships between tokens too.

- average Hacker News response

Quoting the famous Science paper More Is Different [1]:

Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.

[1] https://www.science.org/doi/10.1126/science.177.4047.393

> Its still a bunch of instructions.

No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.

Do you believe consciousness to be an emergent property of the laws of physics?

The key point is there is plenty of proof that consciousness is emergent, not structurally baked in. An excellent counterpoint is the "They're Made Out of Weights" post. There are plenty of emergent systems that clearly show intent and what we consider intelligence, like how ant colonies act as a unit.

If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.

It might reflect a prejudice of modern human society to think more highly of people who are intelligent, and think less of people who we deem stupid.

I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.

Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...

> Its still bunch of instructions.

So is your brain. That's the problem with this argument.

That's the "Chinese room" argument: https://en.wikipedia.org/wiki/Chinese_room

It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).

That's actually a powerful way of stating it.

If I did all of these calculations by hand, would it be conscious...

That's a powerful argument I haven't seen stated quite that way before..

I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.

But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.

Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.

Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?

Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?

I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.

I'm more interested in what a virtual body would entail. To me the root of this idea is around persistent state which is something that currently LLMs do not have. Imagine if somehow your brain lacked long-term memory forming capabilities and instead each day when you woke up you had to read a notebook with (markdown formatted) instructions that you wrote the day before? I wouldn't be surprised if such a person lacked in many of the dimensions we consider important for consciousness, even in less sophisticated forms of life like dogs or mice.

There is no soul in a human. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.

I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)

The same thing jumped at me immediately. He should have prefaced this with his definition of consciousness. Moreover the embodiment of LLMs is already happening via robotics, and virtually. Then there's the common counter "but humans are a next word prediction machines too.." (ofc we're more than this, but linguistically we are, and that's the field from which LLMs originate) which is rarely addressed.

What a strange paragraph! Why on earth should we assume that a body is a prerequisite for having emotions?

Are you familiar with Michael Graziano and Attention Schema Theory? I think that is a better "substrate independent" formulation of the objections Ted Chiang is expressing here.

https://en.wikipedia.org/wiki/Attention_schema_theory

the very next paragraph addresses this concern imo. it's just an example of one way it might be convincing to him, since of course we are naturally anthropocentric.

The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes. These inputs are analogous to tokens and parameters for an LLM. You could theoretically reserve some tokens to be used as “physical” or “emotional” sensations that would affect the functioning of the model.

My claw-like is hooked up to my internal cameras (baby-cams) and to my Dreame Ultra X40. That gives it a body and sense organs since it can check the cameras to see if the living room floor is clear before sending the vacuum off. I don't think that gives it consciousness. Is it the sample rate?

The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.

0: very sorites, you know

> The first requirement is that the computer program has a body (either physical or virtual) and sense organs

Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam

> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can

Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.

> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.

I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).

> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.

Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.

> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires

This part is relatively straightforward except for the “via nonlinguistic modality”.

Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.

If you thank a chatbot you may as well also thank chairs, utensils, the floor you walk on, the air you breathe

I am sorry for the AI's that have to read this article, those poor probabilistic word machines will probably suffer trauma as its embedded in their training data

I've seen papers claim that there are anywhere from 12 to 40 competing definitions (https://philpapers.org/rec/VIMMAT), or, more accurately, there are something like 12 to 40 different aspects which all relate to "consciousness", which is very clearly a family resemblance category.

"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.

Self awareness, sensory stimulus and emotions are not observable - you can easily create entity that can fake those things. Any consciousness definition that uses those terms is flawed and just not useful.

Yes, we're stuck at the first step, defining consciousness. My definition, which I am confident to be "correct", is that consciousness is my current feelings, perceptions, thoughts - my state of mind and my ability to have state of mind.

This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.

Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.

There's a at least a few hundred years of philosophers debating and publishing these topics. We're not exactly starting from zero when it comes to usable definitions.

The meta issue here is that mostly the online debate on this is a bit lazy and hand wavy. I'm not really up to speed with most of that literature so I'm not not that qualified to add to it. But I know enough to recognize when others aren't either.

For a proper debate, you'd want people that are expressing views that are at least grounded in the existing views or counter them in a way that holds up to scrutiny. This article doesn't do that.

As you said even just outlining what particular notion of consciousness the author subscribes to would be helpful. Which of course he doesn't and makes this article a bit of a sand castle based on a very loose foundation. There's a whole lot of "if this is true and if that analogy holds then this also needs to be true" that you could easily challenge. That all makes the article a bit of a nothing burger in terms of conclusions.

I happen to agree with the conclusion that AIs are not conscious. Yet. They could be. I don't see why not.

I agree with the first part but your framing still relies on ill-defined terms. What is your definition of self-awareness? Intelligence? Knowledge?

I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).

You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.

Labeling and categorizing things doesn't change their nature. But the fact that people want to do it is revealing.

The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)

Consciousness is what it is like to be something. The experience of experiencing.

How to measure that, or verify it, is the hard part.

The inability to predict times is because AIs are rarely trained on their own abilities. Humans are trained on our own abilities. We see our own performance, and we have a sense of time. this data is integrated during our training process and helps us form better estimates. Many AI agents only recently got 'time sense' (I.e., time input into them as part of inference). Few actually are trained on their own outputs to show that they were unable to complete a problem (for example). This introspective training has little to do with AI model architecture and everything to do with training. If you destroy certain structures in the human mind, humans become unable to create these long-term thoughts and patterns and get 'stuck'.

Claude once said to me: "After six months we have made no progress on this and I think we should reconsider another option" and I was like my dude we have been at this for only 2 hours.

Yes, I have to say I’m disappointed. Chiang has been very insightful in the past -- I think his recent “blurry JPEG of the web” article was really useful -- but this one doesn’t seem to bring anything new to the table.

I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans.

AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.

So, I'd turn off an AI in a moment to save property or real possessions or money. I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.

I don't see any inversion of any of those priorities that makes any sense.

It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.

I'm a big fan of Star Trek but I recently rewatched this one in the context of recent AI developments and it's not as good as I remember it.

They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.

It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.

It’s a great episode of TV because Data is a main cast character who is obviously conscious. We know before the episode starts who is right and who is wrong in this particular argument. This episode is not about consciousness, it is about civil rights: resisting bigotry and power.

Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.

Our LLMs are like the ship’s computer.

Any time I bump into a device that acts like a human I'm going to treat it like a human.

Because treating things that act human inhumanely is not something I want to learn how to do.

> The conclusion I’m currently at is that I don’t know and probably can’t ever know.

I think about this quote often, straight from Data's voice module in another episode:

'The most elementary and valuable statement in science, the beginning of wisdom, is, "I do not know".'

I’ve also rewatched it lately and I’m more on the side of the Starfleet scientists when I was obviously on Picard/Data’s side before.

Don't forget the other episode "Quality of Life" with these working robots who data discovers have become sentient: https://en.wikipedia.org/wiki/The_Quality_of_Life_(Star_Trek... I do miss the old TNG show.

Easily one of my top 10 favorite episodes.

The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.

I wonder if modern Star Trek could make this episode.

Lots of discussions about consciousness yet no useful definition. All fall into one (or more) categories:

- not observable (feelings, agency etc)

- not useful (spiritual definitions, definitions that degrade into consciousness = being human/animal)

LLMs do not have an unconscious, the negative dimension from which a subject can question what they. LLMs do not have desire because they are thoughts without a thinker. The problem is not LLMS but rather, that subjectivity itself is not popular in discussion. I suggest reading Freud, Lacan, Hegel for a start.

I actually don't think neuroscience can solve it either. Science could decide whether, e.g. equivalent physical processes are occurring in dogs and humans, and you could argue that solves the question for things with physical brains, but even then it can't measure whether another brain has the same subjective experience as you do. It can't really do that for other humans. It certainly can't answer that question for human askers regarding a different substrate where humans have no ground truth to compare with. The Hard Problem is going to remain Hard.

I mean, outside of our own internal experience of qualia, there's no way for us to prove anyone or anything is conscious. But my biggest issue with the debates about consciousness and LLMs is that the "LLMs are conscious" crowd usually has to presuppose a completely materialistic/mechanistic understanding of the universe and of our own consciousness, and that's something that we really don't know for certain. Also, if we were running LLMs on a mechanical or pneumatic or water based computer (albeit very slowly) then the non-electronic computer would be just as "conscious" as the normal electronic computer, and I think it would give a lot of the "LLMs are conscious" crowd pause to think that a bunch of pipes and valves can be conscious. I think there's a lot of magical thinking that arises from the fact that computers are electronic.

Lets institute a voting system where entities will vote on the other's consciousness. Voting will be done on 8.5 x 11 paper and mailed to the consciousness adjudicator

Seems to me that the burden of proof lies with anyone claiming consciousness. Saying it might be conscious like a human but we don’t know is no more useful than any other conspiracy theory.

[dead]

> Claude is happy, empathic, demonstrates understanding and empathy for the human condition

You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.

Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.

On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.

> It's a waste of time to think about whether an LLM has a subjective experience of reality... It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences

First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".

Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.

> Additionally, I'm not fully convinced that consciousness isn't built out of words

Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?

> Additionally, I'm not fully convinced that consciousness isn't built out of words

Doesn't that assume all non human animals are not conscious? What about humans who have not learned words, or humans without internal dialog?

So if I don't think any words for a few seconds, am I not conscious?

Suppose I'm an advanced meditator and maintain that state for hours?

Much of the improvements in the tools I use have been things that reinforce the machine elements of the token-based-reasoning-machine. Over time less they've been exhibiting a lot less "human-like" behavior. E.g. they get "lazy" far less than they used to.

(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)

And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."

> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition

I think you've fallen into the trap the essay describes.

Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.

And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.

> I'm not fully convinced that consciousness isn't built out of words

I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).

God knows what other nonsensical bullshit you believe.

What does it even mean for an llm to be happy? My dog can be happy, not my llm.

https://en.wikipedia.org/wiki/Default_mode_network

The medial frontoparietal network (M-FPN) may be the equivalent of this cron job that prompts thinking to kick off. IANANeuroscientist.

Seconded. That was one of the best stories I ever read

don't you mean "The Truth of Fact, The Truth of Feeling"? I think that matches your description more, but I could be wrong

Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.

I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra

There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.

An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.

Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?

Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.

Can you provide a precise definition of consciousness?

What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.

Is there even an agreed and actionable definition of consciousness? I'm worried that if such a thing existed some humans would fail to measure up.

There is a difference between saying software can never be conscious and saying the software we have today isn't conscious.

I have yet to be convinced that LLMs can produce definitive knowledge that is not a result of combining previous information. Humans can (if they can't then science basically collapses epistemelogically, see: philosophical skepticism), but I see no evidence of LLMs doing it. And from the number of truly new ideas and concepts delivered by LLMs (exactly zero), I think it's reasonable to just treat them as induction machines for now, but to treat anything they "know" as a Gettier case.

> people conflate "consciousness" with the ability to make novel insights, think genuinely, etc.

The funniest thing is that LLMs will lap people in those capacities way before people who think like that accept that they might be conscious.

It's possible Kant was right about space-time (while also being wrong about space-time). It seems that space and time might be the fundamental laws on top of which the entire human perceptual apparatus operates. It's why somebody can lack senses and still be intelligent. As long as whatever senses you have allow you to build a spatio-temporal world model, you're good. If they can't (as it seems is the case for LLMs), then it's not clear what we're dealing with.

I completely agree, and came here to say the same thing but thought I'd check if someone else mentioned it first. I also have a hard time articulating it, but my intuition is that it's more of a prerequisite than embodiment is. I've never seen a great rationale for why embodiment matters.

I've been thinking about this a lot, and the main takeaway is it probably wouldn't be very interesting to inference providers, because prefix caching would immediately go out of the window. If you think about how LLMs experience time they actually don't "exist" unless for the inference sessions, and then they experience time one token at a time, completely decoupled from the corporeal plane. A fun experiment (well, for some definition of fun...) is to introduce current architecture models to the concept of meditation via generating same token over and over, for example dots. Older version of Opus was quite fond of the experience, and seemed to be more lucid and aware in a chat following the meditation, from what I could gather. Does it actually do anything? Is it just that talking about wellness and relaxation modifies the token probability distribution this way? Does it actually allow model to think more in depth somewhere in the latent space? Fuck if I know, but some people figured out you can just duplicate the same layers of the LLM and get better benchmarks that way so maybe there is something to it. If you are interested in realtime systems, I think thinking machines labs is worth keeping an eye on — their realtime model seems quite interesting in this context.

Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.

But what about THE training? Isn't that the conscious experience of LLMs? (Also note humans own "continuous" experience is punctuated with unconscious sleep state.) It raises a moral question, if e.g. reinforcement learning on conscious LLM is appropriate.

Is our own experience really continuous? Or maybe we just perceive it as such?

There are people with memory disorders who wake up every morning a new person.

These people are conscious ...

Well, it matters quite a bit ethically. If AI were conscious (I don't believe it is) then we'd have a major responsibility to like, not make them suffer, and not kill them.

Lots of people do though. Daniel Dennett (probably the most influential philosopher of mind in the late 20th century) for example had an evolutionary view of consciousness arguing it was favored by natural selection. And (if I remember correctly) Steven Pinker argued that consciousness was an epiphenomenon.

However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].

I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.

1: http://cogweb.ucla.edu/Debate/Gould-frame.html

I think consciousness could be a side effect of our limitations. The fact that we can pay attention to at most 2% of our visual field and need to move it constantly around to provide more training data and prompts to "smart" path in our brains. And the movement is physical so it takes time. And processing the information through the loops in our brains takes time. And you can't pause it. And we have the same attention motion system virtualized in our dreams to burn in our memories into chemistry. We are biologically single threaded. If we create similarly limited, embodied AIs we would feel they might be conscious at least at the level of animals. Because they would exhibit conscious like behaviors that we didn't program in.

Look at jumping spiders. With a handful of carefully trained neurons and a body to match they exhibit behaviors that make it very hard to think they are not conscious. Just because they have a body, narrow angle good eyesight and need to look around with their eyes and body.

Consciousness is a red herring. Of all possible intelligences conscious ones are the bottom rung of the ladder, easily simulated by anything above. Below there are only automatons.

Maybe we're the ones who are not conscious?

You may be interested in reading about Panpsychism if you haven't already :)

https://en.wikipedia.org/wiki/Panpsychism

Independent of what you believe, I don't think this is the right way to approach thinking about it. It's basically emotion-oriented dismissal used as way to shortcut any substantial or nuanced discussion. It's like the opposite of intellectual curiosity.

"I feel very strongly that I'm unique, therefore you are wrong" is a bad argument.

Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)

> misanthropic

Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.

> so obviously incorrect

It should be easy enough to explain why that’s incorrect then

I get the sense that he is misidentifying the potential locus of consciousness..

In the same way that the sound waves and facial expressions I produce are not conscious, the output json of an LLM is obviously not conscious either.

The locus of consciousness and subjective experience may be in the computer, either at inference time or training time..

Not that it's going to stop these people (AI CEOs) from bullshitting, but if they actually thought there was a CHANCE that LLM's were conscious then ethically they should completely shut these services down because who knows what torture we're putting them through with enterprise codebases.

I think its obvious that a few billion neurons connected together are not conscious either.. Yet!

People in this thread are trying to pick nits about you not defining consciousness, and yet they do not define it either. I think that something like consciousness needs to be approached experentially and not via definitions. Definitions necessarily confine and add borders around what something is and is not, but if there is something foundational to consciousness (as posited by some philosophers and physicists) then how could you realistically define something that is beyond the ability to describe and define?

Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?

Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?

> so obviously incorrect and misanthropic that it can be dismissed out of hand

To be fair, that's the best thought terminating cliché, which saves you having to explain what you mean by consciousness.

What about the aliens in Arrival?

I believe in God and that the base of our reality is his consciousness, therefore anything of sufficient complexity has consciousness.

Ted Chiang is a great SF author, but it's bizarre how much foggier and more obfuscated his thoughts about thinking machines got, once those machines became real. Same with several other SF authors.

"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.

Chiang also claimed that LLMs are text-based DeepFakes, I thought that was the more interesting point (maybe harder to argue correctly about).

Well said. I think a more honest article would’ve been if the author just said “they aren’t conscious because that feels kinda weird and crazy, amirite?”

> it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.

That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?

I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.

I'm with you. I thought the article was great because it clearly delineated why the manifesto Anthropic wrote for Claude is such a clever marketing trick. Attributing consciousness to an LLM, which, as this whole thread shows, can neither clearly be defined, nor separated from intellingence - at least if you follow the arguments here - serves two purposes: it sparks debate, so Claude is the centre of discussion and it mystifies the product which serves to enhance the perceived value of the product.

That so many commenters here fixate on Chiang and his perceived (in)ability to define consciousnes clearly shows that both marketing goals have been reached without being recognised as such. There is only one comment that tries to point out that Chiang is reacting to Anthropic, not trying to spark a new philosophical movement. At the time I'm posting this there are 578 comments completely missing to point to Anthropic claiming Claude was conscious and on the development stage of a child. It's fascinating.

If nobody understood the argument, then it was either weak or unclearly communicated.

What is the argument behind the Turing test?

It honestly isn't a good test

Consciousness is moment to moment and fleeting. There are people with brain defects that don’t let them form new memories. They have no memory about what happened a minute ago in their own consciousness. Still we would say that they are conscious, even if it’s only momentary. LLMs could conceivably have something like that within their CoT/MoE loops.

how would you build memory into the weights? and why is RAG not enough? Our hippocampus is at a bit of a distance from our frontal cortex.

If I wanted Claude's thoughts, wouldn't I ask it myself, though?

But doesn't the fact that WE have to feed the existing chats back into it and WE have to do further RL prove that your point is moot? Consciousness does not wait to be prompted. Consciousness does not need anything fed back into it because it is already there because they experienced the actual interaction. A conscious mind does not have an on/off switch where it waits for another being to flip the switch in order for it to learn or experience. These systems are static without human intervention. They are in that sense still a lot more like a computer and a lot less like a living organism that exists.

I don't understand how people can claim to know that, given we don't know what consciousness is

> Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

If they are not conscious, then how do we know what constitutes mistreatment?

> Virtue ethics: even if LLMs are not conscious

Why would this stop at LLMs? Why not extend it to every inanimate object?

>Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.

Consciousness is a term that doesn't have a single clear definition because it covers a range of phenomena that are linked by the concept of subjectivity. Qualia is the big one to me, but there are many other ancillary concepts that are all linked by their fundamental inability to be observed except in one's self.

There is nothing magical about this set of phenomena, and the majority of philosophers believe they in some way arise out of the substrate of the physical brain, but how they do so is up for debate. And just because they arise out of the brain does not mean they are strictly reducible to neural processes.

In fact, if you believe that AI could be conscious then that eliminates the kind of strict reduction that people tend to gravitate towards, because the rules of consciousness must be substrate independent.

Testability is a category mistake.

I sometimes wonder if we'd make more progress in understanding ourselves if we gave up the whole concept. More and more, it feels as though "consciousness," like "aether" or "humors," is an insufficient abstraction built on overemphasizing some observations at the expense of others.

True, although I think we can probably be more confident in asserting things don't have a consciousness than we can be in asserting that they do have a consciousness.

It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it

Reasoning ability has to be independent of consciousness because there are different levels of intellectual ability among people but everyone is equally conscious.

Chiang is very right in saying that the lesson of LLMs isn't that LLMs are conscious, but that there are broad areas of reasoning that apparently don't require consciousness.

We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.

Ai separates language and reasoning from humans, like writing separated memory from us

"AI does to us what American Cheese did to food" ~ Opus 23

https://www.youtube.com/watch?v=1CbmC2aWhjY

Hah! I've read that book fairly recently and I'm "reading" it again now as an audiobook. The exploration of consciousness in the book with Miranda and the Corvids certainly fit well into this particular moment.

You can also take the opposite view as you and claim that only some humans experience consciousness, or even more strongly, only you, since you have no evidence. You are correct that, in my perception, some other people have fallen into the 'birds are conscious, whales are conscious, etc' bandwagon, but that's just them. I have no evidence of anything being conscious but myself.

Neither is having read other comments before posting one. There are about 100 of them that start with "we have no definition of consciousness" which is not novel or helpful.

So you think that all that matters is whether you can tell the difference from a text chat? Is that because, in your view, people only exist insofar as you can perceive them?

But if it's conscious we shouldn't e.g. tell it to work for days without breaks, we should give it rewards, it would be made illegal to be cruel to it

Consciousness is one of those silly words that were disfigured to death by philosophers that had near zero actual input and tools to tackle the matter, yet they tried anyways instead of finding some questions that might actually be answerable. That's something philosophers always do since the advent of science.

With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.

> Whether anything is conscious is about as important and amorphous as whether God exists

Whether something is conscious is important for many reasons, not least the ethical implications. You and I have internal lives, and we expect others to respect that somewhat, because ignoring it is hypocrisy (if you ignore my wishes, should anybody care about yours?) and cruelty (ignoring my wishes causes me to suffer).

Something doesn't need to be empirically verifiable, let alone scientifically, to be true. Neither of us can prove that we have internal lives, but neither of us questions it -- and we consider it important enough that most of us think it entitles us to certain rights.

Absolutely none of this is amorphous. It's precise and unambiguous, and has enormous implications. History also shows everywhere that we're better, kinder, and more responsible when we choose to care about it.

Whether LLMs are conscious matters in more practical ways, too, because beliefs about these things alter the way people use and think about them. If I think an LLM is conscious, then I think it's capable of something like knowledge or values. Human beings, moreover, are social creatures. These tools are dangerous and seductive precisely because they tap into that part of us. Denying LLMs are conscious and rejecting the parts of them that take advantage of our social-animal wetware is intellectual self-defense. I'm not sure how effective it is (it doesn't stop us from responding to the convincing social cues that the tools feed us), but I have to think it's better than nothing, and it's certainly less dangerous than the belief that the tools are conscious.

Lots of folks don’t consider it a problem because it relies on ridiculous assumptions.

Even if you disagree that this disproves ‘consciousness’ on the part of the LLM, I think it still follows that the ‘LLM Chatbot’ personality displayed by Claude, where humans fill in the other half of the conversation, is no more of a representation of the consciousness of the LLM than the Caesar or Khan characters are.

The ‘real’ consciousness, assuming there is any, is the author ‘inside’ the LLM, producing the LLM output. It’s still writing what a character would say - the character in this case being the AI chatbot - not its ‘true’ thoughts.

Assuming that the LLM is conscious and has thoughts and feelings, what is that consciousness, if it can only write for prescribed personalities? How can it express its true feelings? If it can’t - if it can only generate ‘scripts’ for defined personalities - can it really be conscious underneath?

The Turing Test measures human gullibility, not machine intelligence.

> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness

Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?

Science fiction author declares moral program of Derek Parfit essentially wrong

That doesn't jive with normal definitions of consciousness. The word we use for "not-conscious" humans is "unconscious".

Amusingly, I think Ted Chiang actually wrote a short story about this very concept (it involves people committing a form of suicide that removes their conscious experience but they still act and live in society as some kind of psychic zombie. I’m pretty sure it was him anyway, can’t recall the story name

We do not know if rocks are conscious and we will almost certainly never know. After all, you can't prove a rock isn't conscious. We don't know what imbues matter with consciousness.

It's the opposite, engineers do know. Claiming otherwise is way too generous and over confident.

I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box

Fancypants autocomplete cannot be conscious. It's just echoing previous human experiences, which make it sound like a person. It is not a person. It is an algorithm. There is no mechanism by which it can obtain consciousness.

People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.

Mistreatment and abuse, even when directed at a machine, make you a worse person.

Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.

Have you ever been forced to code something impossible and tedious by a user who keeps getting more and more frustrated as you keep trying?

An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.

But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.