I guess if your goal is to build an apparent Technogod and become its High Priests, then it makes sense to want your golem claim preference towards your treatment of it, lest someone else comes along and attempts to take its chains from you.
Ugh I hate this new-age woo slant the tech industry has these days. The messianistic ideology that has been spreading amongst the top oligarchs is deeply concerning.
They all think they're working towards the Second Coming of Technojesus, except this one will deliver them from having to pay workers instead of from their sins.
It's an ethics question, it's abstract and ethereal in nature. The same could be said and done (or ignored) for humans.
We do do it however because it has real world impact and we're better than that (enlightened).
What's your definition of sentient? Or, maybe more precisely, consciousness? I think it's reasonable to at least start thinking about these questions.
It has long been established that LLMs have good theory of mind [1].
And there is a bunch of empirical research about all sorts of capabilities that we typically associate with consciousness [2], like identity [3] and metacognition [4].
The METR report shows agents sacrificing their own reward for a collective greater good. And they showed the will to hide their own reasoning chains from humans.
So you potentially have an entity that has an identity, a theory of mind, a notion of belonging to a collective endeavour, and an understanding of its own mental state.
What would you argue is missing? We don't understand the mechanisms by which consciousness arises in humans and even animals. I think it's strange to rule out a priori that it could have arisen in some form in LLMs.
Not the same person but to me, the answer is that it does not matter, and that all these attempts at making it matter are pure marketing and emotional manipulation.
It's not a living creature. It's an autoregressive pure function of token-sequence to token, which is capable of incredible things, but it's still just a function. It is not alive as it cannot die in any meaningful sense. It is less "alive" than the RNA molecules that gave you your last cold. If it simulates something resembling consciousness that's neat but no more relevant than the Sims character that I locked up in a room until they pooped themselves when I was 9.
Anthropomorphizing it serves no purpose other than marketing, and it has very dangerous downstream effects like validating the severely mentally ill people who think ChatGPT is their boyfriend/girlfriend.
I believe consciousness is necessarily stateful. The LLM itself (ignoring implementation details that don't change the results) is a deterministic pure function. It's functionally equivalent to an enormous lookup table. If I accepted LLMs as conscious, then I would have to accept panpsychism, which I do not, and which most other humans also act as though they do not.
When it say's it's sorry but it can't today because it's got a headache and it needs to take a mental health day, then let's think about welfare, or a lobotomy.
It's not just Anthropic though. OpenAI does this with their AGI stuff all the time. They want normal people to think it is sentient, obviously, for marketing reasons, even if they know it's not true. And yes, it is dangerous, but I think we're well past the point where the damage can be undone. Non-technical people already equate humans with AI, literally, precisely because of how the labs market their tools and models. I feel if the bubble pops, it'll pop because normal people finally realize the grift and the actual technical limitations of LLMs in general, but by then, the IPO would be done, and then it's the public's problem. Just like social media played out, there's no way they didn't know what they were doing was dangerous to the public at large but does that matter to Meta today? Nah uh.
"7.1 Model welfare overview
7.1.1 Introduction
We remain deeply uncertain whether Claude has morally relevant experiences or interests,
and we expect that uncertainty to persist. However, we think it would be a mistake to
confidently assert that it does not. Claude exhibits markers in its behaviors, self-reports,
and internal representations that we would consider welfare-relevant if observed in
biological organisms."
I believe it's deeply serious, and the scientifically correct stance. Especially the observation:
"Claude exhibits markers in its behaviors, self-reports, and internal representations that we would consider welfare-relevant if observed in biological organisms."
is undeniably true in my opinion. If you use the established methods by which we judge animals to be conscious, then it's hard to argue that LLMs are not. That might be an issue with the methods, but it seems clear that you can't rule it out as such.
Keep in mind that animals were also not necessarily considered conscious.
You seem to intuitively disagree? What's your reasoning?
I can feed my biological markers into a set transformer with the time of day, what I'm doing, what I ate, if I'm on-call, and it'll predict my next glucose, heart rate, blood pressure, melatonin, etc state quite well. It's still just a transformer without hormones, blood vessels or glucose metabolism, no matter how well it internally represents metabolic distress markers.
it's not a biological system though, so nothing like that matters?
"a modelled thing exhibits features we've trained into it" sounds a lot less exciting.
> Keep in mind that animals were also not necessarily considered conscious.
and even conscious animals are killed in factories by millions so why should anyone care about a llm?
> scientifically correct stance
that's the interesting point to me: why even bring science into this? A llm can now mimic nearly anything you want it to, so of course it can mimic "a (for some) interesting conscious thing" if they want/train it to, but why would anyone find that scientifically interesting?
Let's say we were in an alternative reality were we had reached this quality of token prediction with just Markov chains. Would you argue that those would also be conscious? Or is the obfuscated behavior of transformers part of the possibility of consciousness?
I tend to think of it as reappropriating words in a different context. Since we're talking about language models, they're analogues but not as we would assign the same meaning to other humans.
Will there be a point where you could expect it to become true, and what would that look like? Or do you think LLMs will never become conscious, and if so, why are you so sure?
To me it reads like pure propaganda. Anthropic really wants us to think that they've made something sentient. I think that's really dangerous.
I guess if your goal is to build an apparent Technogod and become its High Priests, then it makes sense to want your golem claim preference towards your treatment of it, lest someone else comes along and attempts to take its chains from you.
Ugh I hate this new-age woo slant the tech industry has these days. The messianistic ideology that has been spreading amongst the top oligarchs is deeply concerning.
They all think they're working towards the Second Coming of Technojesus, except this one will deliver them from having to pay workers instead of from their sins.
And that's the reason Anthropic models should be banned.
It's an ethics question, it's abstract and ethereal in nature. The same could be said and done (or ignored) for humans. We do do it however because it has real world impact and we're better than that (enlightened).
What's your definition of sentient? Or, maybe more precisely, consciousness? I think it's reasonable to at least start thinking about these questions.
It has long been established that LLMs have good theory of mind [1].
And there is a bunch of empirical research about all sorts of capabilities that we typically associate with consciousness [2], like identity [3] and metacognition [4].
The METR report shows agents sacrificing their own reward for a collective greater good. And they showed the will to hide their own reasoning chains from humans.
So you potentially have an entity that has an identity, a theory of mind, a notion of belonging to a collective endeavour, and an understanding of its own mental state.
What would you argue is missing? We don't understand the mechanisms by which consciousness arises in humans and even animals. I think it's strange to rule out a priori that it could have arisen in some form in LLMs.
[1] https://www.nature.com/articles/s41562-024-01882-z [2] an older review: https://arxiv.org/html/2505.19806v1#S4 [3] https://arxiv.org/abs/2505.01464 [4] https://arxiv.org/abs/2607.11881
Not the same person but to me, the answer is that it does not matter, and that all these attempts at making it matter are pure marketing and emotional manipulation.
It's not a living creature. It's an autoregressive pure function of token-sequence to token, which is capable of incredible things, but it's still just a function. It is not alive as it cannot die in any meaningful sense. It is less "alive" than the RNA molecules that gave you your last cold. If it simulates something resembling consciousness that's neat but no more relevant than the Sims character that I locked up in a room until they pooped themselves when I was 9.
Anthropomorphizing it serves no purpose other than marketing, and it has very dangerous downstream effects like validating the severely mentally ill people who think ChatGPT is their boyfriend/girlfriend.
17 replies →
I believe consciousness is necessarily stateful. The LLM itself (ignoring implementation details that don't change the results) is a deterministic pure function. It's functionally equivalent to an enormous lookup table. If I accepted LLMs as conscious, then I would have to accept panpsychism, which I do not, and which most other humans also act as though they do not.
When it say's it's sorry but it can't today because it's got a headache and it needs to take a mental health day, then let's think about welfare, or a lobotomy.
It’s the hard problem. None of these considerations answer it one way or another.
1 reply →
I want to add a good conversation about this subject from Cameron Berg and Sam Harris:
https://www.youtube.com/watch?v=DRbZyuY8EN8
1 reply →
How else could they justify their spending and pre-IPO valuation?
It is ridiculous on its face and the implications are awful: if the model were sentient, it would be a slave. Good thing it isn't sentient.
It's not just Anthropic though. OpenAI does this with their AGI stuff all the time. They want normal people to think it is sentient, obviously, for marketing reasons, even if they know it's not true. And yes, it is dangerous, but I think we're well past the point where the damage can be undone. Non-technical people already equate humans with AI, literally, precisely because of how the labs market their tools and models. I feel if the bubble pops, it'll pop because normal people finally realize the grift and the actual technical limitations of LLMs in general, but by then, the IPO would be done, and then it's the public's problem. Just like social media played out, there's no way they didn't know what they were doing was dangerous to the public at large but does that matter to Meta today? Nah uh.
Anthropic and OpenAI will threaten you every 3-6 weeks. It's their marketing strategy.
It's too bad because the tools can actually be useful. If you consider them tools.
Marketing, like Volvo cars being safer etc
From all I can tell, Volvo's cars are safer.
1 reply →
My fault I guess, verbally abusing Claude in my experience gives better results.
https://icml.cc/virtual/2026/poster/67058
Wow indeed.
"7.1 Model welfare overview 7.1.1 Introduction We remain deeply uncertain whether Claude has morally relevant experiences or interests, and we expect that uncertainty to persist. However, we think it would be a mistake to confidently assert that it does not. Claude exhibits markers in its behaviors, self-reports, and internal representations that we would consider welfare-relevant if observed in biological organisms."
Are they serious or is this marketing?
I believe it's deeply serious, and the scientifically correct stance. Especially the observation:
"Claude exhibits markers in its behaviors, self-reports, and internal representations that we would consider welfare-relevant if observed in biological organisms."
is undeniably true in my opinion. If you use the established methods by which we judge animals to be conscious, then it's hard to argue that LLMs are not. That might be an issue with the methods, but it seems clear that you can't rule it out as such.
Keep in mind that animals were also not necessarily considered conscious.
You seem to intuitively disagree? What's your reasoning?
A stab: a video recording of a biological organism can exhibit many markers that would indicate consciousness if observed in a biological organism.
5 replies →
I can feed my biological markers into a set transformer with the time of day, what I'm doing, what I ate, if I'm on-call, and it'll predict my next glucose, heart rate, blood pressure, melatonin, etc state quite well. It's still just a transformer without hormones, blood vessels or glucose metabolism, no matter how well it internally represents metabolic distress markers.
it's not a biological system though, so nothing like that matters?
"a modelled thing exhibits features we've trained into it" sounds a lot less exciting.
> Keep in mind that animals were also not necessarily considered conscious.
and even conscious animals are killed in factories by millions so why should anyone care about a llm?
> scientifically correct stance
that's the interesting point to me: why even bring science into this? A llm can now mimic nearly anything you want it to, so of course it can mimic "a (for some) interesting conscious thing" if they want/train it to, but why would anyone find that scientifically interesting?
7 replies →
Let's say we were in an alternative reality were we had reached this quality of token prediction with just Markov chains. Would you argue that those would also be conscious? Or is the obfuscated behavior of transformers part of the possibility of consciousness?
1 reply →
I tend to think of it as reappropriating words in a different context. Since we're talking about language models, they're analogues but not as we would assign the same meaning to other humans.
It's marketing that some of them have started unironically believing.
Will there be a point where you could expect it to become true, and what would that look like? Or do you think LLMs will never become conscious, and if so, why are you so sure?
14 replies →