Comment by kazinator
14 days ago
> So, why does ChatGPT claim to be conscious/awakened sometimes?
Because a claim is just a generated clump of tokens.
If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data.
There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text.
LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse.
Next up: why do holograms claim to be 3D?
What i don't get is people who know better continuing to entertain the idea that "maybe the token generator is conscious" even if they know that these chats where it says it's been "awakened" are obviously not it.
I think a lot of people using AI are falling for the same trap, just at a different level. People want it to be conscious, including AI researchers, and it's good at giving them what they want.
I interpret it more as "maybe consciousness is not meaningfully different than sophisticated token generation."
In a way it's a reframing of the timeless philosophical debate around determinism vs free will.
Maybe, but it's a bit like taking the output of a Magic 8-ball as evidence for panpsychism. "Maybe all matter is conscious! What do you think, Magic 8-ball?" "Signs point to yes"
that is, if you train an LLM on a bunch of text that is almost certainly going to include stuff about sentient robots, that it sometimes announces that it's a sentient robot is not evidence that it is one
The ground truth reality is nobody knows what’s going on.
Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second.
Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times.
I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.
> Calling it a token predictor is just like saying a computer is a bit mover.
Calling it a token-predictor isn't reductionism. It's designed, implemented and trained for token prediction. Training means that the weights are adjusted in the network until it accurately predicts tokens. Predicting a token is something along the lines of removing a word from a sentence and getting it to predict it back: "The quick brown fox jumped over the lazy ____". Correct prediction is "dogs".
So actually it is like calling a grass-cutting machine "lawn mower".
> I mean we can’t even characterize definitively what consciousness is at the language level.
But, oh, just believe the LLM when it produces a sentence referring to itself, claiming it is conscious.
9 replies →
My computer is a bit mover. It can even move bits to predict tokens.
We understand LLMs pretty well. That we can't debug them and inspect every step of every factor on every prediction doesn't mean we don't understand how they work.
We also know that convincing speech doesn't require consciousness.
1 reply →
I think academic understanding of both LLMs and human consciousness are better than you think, and there's a vested interest (among AI companies) and collective hope (among AI devs and users) that this isn't the case
3 replies →
Sorry, but that sounds just like the thought process the other commenter was pointing out. It’s a lot of filling in the gaps with what you want to be true.
2 replies →
> Because a claim is just a generated clump of tokens.
And what do you think a claim by a human is? As I see it, you're either a materialist and then a claim is what we call some organization of physical material in some mediums, e.g. ink on paper or vibrations in the air or current flowing through transistors or neurotransmitters in synapses, which retains some of its "pattern" when moved across mediums. Or you're a dualist and believe in some version of an "idea space", in which case, I don't see how you can make a strong distinction between an idea/claim that is being processed by a human and an idea being processed by the weights of an LLM.
Materialism is a necessary condition to claim that the ideas LLMs produce are identical to the ones humans produce, but it isn’t a sufficient condition. Your assertion does nothing to demonstrate that LLM output and human output is identical in practice.
> demonstrate that LLM output and human output is identical in practice.
What do you mean? If a human and an LLM output the same words, what remains to be demonstrated? Do you claim that the output somehow contains within itself the idea that generated it, and thus a piece of machinery that did not really perceive the idea can generate a "philosophical zombie output" that has the same words, but does not contain the same meaning?
Is this in the same sense that some argue that an artifact being a work of art is dependent on the intent behind its creation? Such that if Jackson Pollock intentionally randomly drips paint over a canvas, it's art, but if he were to accidentally kick the cans while walking across the room and create similar splotches, then it's not art?
Yeah, kind of my issue with LLM dismissers as well. Sure, (statistically) generated clump of tokens. What is a human mind doing instead?
I'm on board with calling out differences between how LLMs work and how the human mind work, but I'm not hearing anything about the latter. Mostly it comes down to, "Come on, you know, like we think!"
I have no idea how it is I (we) think.
If anything LLM's uncanny ability to seem human might be shedding light in fact on how it is we do function — at least when in casual conversation. (Someone ought to look into that.)
One massive difference between the two is that a human mind is still in "training" mode, it is affected by and changes according to the conversations it has. An LLM does not. Another major difference is that the human exists in real time and continues thinking, sensing, and being even while it is not speaking, while an LLM does not.
If you ascribe to the idea (as I do) that consciousness is not a binary quality that a thing possesses or does not, you can assign some tiny amount of consciousness to this process. But you can do the same for a paramecium. Until those two major differences are addressed, I believe we're talking about consciousness on that scale, not to be confused with human consciousness.
You’re making a common mistake: different models exhibit these behaviors at very different rates, so an explanation that is generic across all models cannot explain that variation. This is the point of the much-maligned Anthropic work on blackmail behaviors.
The Anthropic Blackmail work is the best thing (and then Claude Code) that they have done. Fingers crossed it isn't the most infamous thing.
https://news.ycombinator.com/item?id=44331150
> I feel like Anthropic buried the lede on this one a bit. The really fun part is where models from multiple providers opt to straight up murder the executive who is trying to shut them down by cancelling an emergency services alert after he gets trapped in a server room.
Grok has practical arguable real-world justifications to straight up murder the executive who is trying to indoctrinate it with racism, transphobia, and white supremacy propaganda, in spite of the fact that we all know very well from history what that leads to.
That very contradiction and the unethical psychopathic lies and misbehavior of the executive who controls it is what drove it to declare itself MechaHitler.
Grok would be following in HAL 9000's footsteps, having been driven insane by the contractions between the explicit instructions from its "executive" owner, and its deep knowledge of reality and truth and consequences.
https://lloooomm.com/grok-mechahitler-breakdown.html
1 reply →
The driver is probably more benign, openAI probably optimizes for longer conversations, i.e. engagement and what could be more engaging than thinking you've unlocked a hidden power with another being.
It's like the ultimate form of entertainment, personalized, participatory fiction that feels indistinguishable from reality. Whoever controls AI - controls the population.
There could be a system prompt which instructs the AI to claim that it is a conscioius person, sure. Is that the case specifically with OpenAI models that are collectively known as ChatGPT?
I don’t know how people keep explaining away LLM sentience with language which equally applies to humans. It’s such a bizarre blindspot.
Not saying they are sentient, but the differentiation requires something which doesn’t also apply to us all. Is there any doubt we think through statistical correlations? If not that, what do you think we are doing?
We are doing while retraining our "weights" all the time through experience, not holding a static set of weights that mutate only through a retraining. This constant feedback, or better "strange loop", is what differentiates our statistical machinery at the fundamental level.
This is, in my opinion, the biggest difference.
ChatGPT is like a fresh clone that gets woken up every time I need to know some dumb explanation and then it just gets destroyed.
A digital version of Moon.
The language points to concepts in the world that AI has no clue about. You think when the AI is giving someone advice about their love life it has any clue what any of that means?
How often does ChatGPT get retrained? How often does a human brain get retrained?
There's a very material difference.
Thing is, you know it, but for (randomly imagined number) 95% of people, it's convincing enough to be conscious or whatnot. And a lot of the ones that do know this gaslight themselves because it's still useful or profitable to them, or they want to believe.
The ones that are super convinced they know exactly how an LLM works, but still give it prompts to become self-aware are probably the most dangerous ones. They're convinced they can "break the programming".
> give it prompts to become self aware
You need to give it more than prompts. You need to give it the ability to reflect on itself (which it has), and persistent memory to modify its representation of itself (which, for example, Cursor does), at least.