Comment by MostlyStable
14 days ago
I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.
To be clear, I'm relatively confident that LLMs aren't conscious, but I'm also not so overly confident to claim, with certainty, exactly what their internal state is like. Consciousness is a so poorly understood that we don't even know what questions to ask to try and better understand it. So we really should avoid making confident pronouncements.
Language and speech comprehension and production is relatively well understood to be heavily localized in the left temporal lobe; if you care to know something whereof you speak (and indeed with what, in a meat sense), then you'll do well to begin your reading with Broca's and Wernicke's areas. Consciousness is in no sense required for these regions to function; an anesthetized and unconscious human may be made to speak or sing, and some have, through direct electrical stimulation of brain tissue in these regions.
I am quite confident in pronouncing first that the internal functioning of large language models is broadly and radically unlike that of humans, and second that, minimally, no behavior produced by current large language models is strongly indicative of consciousness.
In practice, I would go considerably further in saying that, in my estimation, many behaviors point precisely in the direction of LLMs being without qualia or internal experience of a sort recognizable or comparable with human consciousness or self-experience. Interestingly, I've also discussed this in terms of recursion, more specifically of the reflexive self-examination which I consider consciousness probably exists fundamentally to allow, and which LLMs do not reliably simulate. I doubt it means anything that LLMs which get into these spirals with their users tend to bring up themes of "signal" and "recursion" and so on, like how an earlier generation of models really seemed to like the word "delve." But I am curious to see how this tendency of the machine to drive its user into florid psychosis will play out.
(I don't think Hoel's "integrated information theory" is really all that supportable, but the surprise minimization stuff doesn't appear novel to him and does intuitively make sense to me, so I don't mind using it.)
Again, knowing that consciousness isn't required for language is not the same thing as knowing what consciousness is. We don't know what consciousness is in humans. We don't know what causes it. We don't even know how human brains do the things they do (knowing what region is mostly responsible for language is not at all the same as knowing how that region does is).
But also, claiming that because a human is anesthetized means they are not conscious is a claim that I think we don't understand consciousness well enough to make confidently. They don't remember it afterwards, but does that mean they weren't conscious? That seems like a claim that would require a more mechanistic understanding of consciousness than we actually have and is in part assuming the conclusion and/or mixing up different definitions of the word "conscious". (the fact that there are various definitions that mean things like "is a awake and aware" and "has an internal state/qualia" is part of the problem in these discussions.)
You said:
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to [produce behavior comparable to that of LLMs], and in what ways those detailed mechanisms are different from what LLMs do.
I addressed myself to those concerns, to which consciousness is broadly not relevant. Oh, conscious control of speech production exists when consciousness is present, of course; the inhibitory effect of consciousness, like the science behind where and how speech and language arise in the brain, is by now very well documented. But you keep talking about consciousness as though it and speech production had some essential association, and you are confusing the issue and yourself thereby.
As I have noted, there exists much research in neuroscience, a good deal of it now decades old, which addresses the concerns you treat as unanswerable. Rather than address yourself further to me directly, I would suggest spending the same time following the references I already gave.
4 replies →
I think that's putting the cart before the horse: All this hubbub comes from humans relating to a fictional character evoked from text of in a hidden document, where some code looks for fresh "ChatGPT says..." text and then performs the quoted part at a human who starts believing it.
The exact same techniques can provide a "chat" with Frankenstein's Monster from its internet-enabled hideout in the arctic. We can easily conclude "he's not real" without ever going into comparative physiology, or the effects of lightning on cadaver brains.
We don't need to characterize the neuro-chemistry of a playwright (the LLM's real role) in order to say that the characters in the plays are fictional, and there's no reason to assume that the algorithm is somehow writing self-inserts the moment we give it stories instead of other document-types.
>once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.
Extraordinary claims require extraordinary evidence. The burden of proof is on you.
I'm not the one making claims. I'm specifically advising not making claims. The claim I'm advising not making is that LLMs are definitely, absolutely not, in no way, conscious. Seeing something that, from the outside, appears a lot like a conscious mind (to the extent that they pass the Turing test easily) and then claiming confidently that that thing is not what it appears to be, that's a claim, and that requires, in my opinion, extraordinary evidence.
I'm advising agnosticism. We don't understand consciousness, and so we shouldn't feel confident in pronouncing something absolutely not conscious.
I agree with your second paragraph, but ...
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.
What is wrong with asking the question from the other direction?
"Explain, in detail and with mechanisms, what it is the human brain does that allows it to do those things, and show those mechanisms ni the LLMs"
We don't know the entirety of what consciousness is. We can, however, make some rigorous observations and identify features that must be in place.
There is no magic. The human (mammal) brain is sufficient to explain consciousness. LLMs do not have recursion. They don't have persisted state. They can't update their model continuously, and they don't have a coherent model of self against which any experience might be anchored. They lack any global workspace in which to integrate many of the different aspects that are required.
In the most generous possible interpretation, you might have a coherent self model showing up for the duration of the prediction of a single token. For a fixed input, it would be comparable to sequentially sampling the subjective state of a new individual in a stadium watching a concert - a stitched together montage of moments captured from the minds of people in the audience.
We are minds in bone vats running on computers made of meat. What we experience is a consequence, one or more degrees of separation from the sensory inputs, which are combined and processed with additional internal states and processing, resulting in a coherent, contiguous stream running parallel to a model of the world. The first person view of "I" runs predictions about what's going to happen to the world, and the world model allows you to predict what's going to happen across various decision trees.
Sanskrit seems to have better language for talking about consciousness than English. Citta - a mind moment from an individual, citta-santana, a mind stream, or continuum of mind moments, Sanghika-santana , a stitched together mindstream from a community.
Because there's no recursion and continuity, the highest level of consciousness achievable by an LLM would be sanghika-santana, a discoherent series of citta states that sometimes might correlate, but there is no "thing" for which there is (or can possibly be) any difference if you alternate between predicting the next token of radically different contexts.
I'm 100% certain that there's an algorithm to consciousness. No properties have ever been described to me that seem to require anything more than the operation of a brain. Given that, I'm 100% certain that the algorithm being run by LLMs lacks many features and the depth of recursion needed to perform whatever it is that consciousness actually is.
Even in context learning is insufficient, btw, as the complexity of model updates and any reasoning done in inference is severely constrained relative to the degrees of freedom a biological brain has.
The thing to remember about sanghika santana is that it's discoherent - nothing relates each moment to the next, so it's not like there's a mind at the root undergoing these flashes of experience, but that there's a total reset between each moment and the next. Each flash of experience stands alone, flickering like a spark, and then is gone. I suspect that this is the barest piece of consciousness, and might be insufficient, requiring a sophisticated self-model against which to play the relative experiential phenomena. However - we may see flashes of longer context in those eerie and strange experiments where people try to elicit some form of mind or ghost in the machine. ICL might provide an ephemeral basis for a longer continuity of experience, and such a thing would be strange and alien.
It seems apparent to me that the value of consciousness lies in the anchoring the world model to a model of self, allowing sophisticated prediction and reasoning over future states that is incredibly difficult otherwise. It may be an important piece for long term planning, agency, and time horizons.
Anyway, there are definitely things we can and do know about consciousness. We've got libraries full of philosophy, decades worth of medical research, objective data, observations of what damage to various parts of the brain do to behavior, and centuries of thinking about what makes us tick.
It's likely, in my estimation, that consciousness will be fully explained by a comprehensive theory of intelligence, and that it will cause turmoil over inherent negation of widely held beliefs.