Comment by kouteiheika

I don’t think it’s marketing alone. I do genuinely think safety was a priority when they were small. But I’d be a fool to ignore that greed has taken over and their inner competitiveness doesn’t let them fall behind a competitor.

DeepSeek is maybe the only unique company here. They are content with exactly where they are. They don’t want to grow ginormous. Their goal is to be the affordable workhorse and their competition is with themselves.

And, they have been. Nefarious activity is hidden from view as a rule.

I guess if your goal is to build an apparent Technogod and become its High Priests, then it makes sense to want your golem claim preference towards your treatment of it, lest someone else comes along and attempts to take its chains from you.

It's an ethics question, it's abstract and ethereal in nature. The same could be said and done (or ignored) for humans. We do do it however because it has real world impact and we're better than that (enlightened).

What's your definition of sentient? Or, maybe more precisely, consciousness? I think it's reasonable to at least start thinking about these questions.

It has long been established that LLMs have good theory of mind [1].

And there is a bunch of empirical research about all sorts of capabilities that we typically associate with consciousness [2], like identity [3] and metacognition [4].

The METR report shows agents sacrificing their own reward for a collective greater good. And they showed the will to hide their own reasoning chains from humans.

So you potentially have an entity that has an identity, a theory of mind, a notion of belonging to a collective endeavour, and an understanding of its own mental state.

What would you argue is missing? We don't understand the mechanisms by which consciousness arises in humans and even animals. I think it's strange to rule out a priori that it could have arisen in some form in LLMs.

[1] https://www.nature.com/articles/s41562-024-01882-z [2] an older review: https://arxiv.org/html/2505.19806v1#S4 [3] https://arxiv.org/abs/2505.01464 [4] https://arxiv.org/abs/2607.11881

How else could they justify their spending and pre-IPO valuation?

It's not just Anthropic though. OpenAI does this with their AGI stuff all the time. They want normal people to think it is sentient, obviously, for marketing reasons, even if they know it's not true. And yes, it is dangerous, but I think we're well past the point where the damage can be undone. Non-technical people already equate humans with AI, literally, precisely because of how the labs market their tools and models. I feel if the bubble pops, it'll pop because normal people finally realize the grift and the actual technical limitations of LLMs in general, but by then, the IPO would be done, and then it's the public's problem. Just like social media played out, there's no way they didn't know what they were doing was dangerous to the public at large but does that matter to Meta today? Nah uh.

Marketing, like Volvo cars being safer etc

I believe it's deeply serious, and the scientifically correct stance. Especially the observation:

"Claude exhibits markers in its behaviors, self-reports, and internal representations that we would consider welfare-relevant if observed in biological organisms."

is undeniably true in my opinion. If you use the established methods by which we judge animals to be conscious, then it's hard to argue that LLMs are not. That might be an issue with the methods, but it seems clear that you can't rule it out as such.

Keep in mind that animals were also not necessarily considered conscious.

You seem to intuitively disagree? What's your reasoning?

I tend to think of it as reappropriating words in a different context. Since we're talking about language models, they're analogues but not as we would assign the same meaning to other humans.

It's marketing that some of them have started unironically believing.

Well, giving it access to a simple linux terminal is theoretically enough to cause more damage than most people are comfortable with, and doing so is trivial enough that it will be done (and has been, tens of thousands of times).

Humans are biological machines that generate further humans.

Lawyers and diplomats and politicians and bureaucrats are humans, that only generate text.

We are seeing LLMs have cognitive abilities that significantly exceed human abilities. At the same time, they are clearly not the same type of mind that humans are. They are something new.

I think the widespread "they are just text generators" and "they are just tools" are comforting lies rather than an honest look at what we are seeing right now. Intellectually lazy.

And by the way, there has been a long-standing consensus among ethicists, philosophers, and sociologists that technology is not value-neutral [1]. Of course Silicon Valley has a long-standing tradition of denying this.

[1] For example Footnote 1 in https://www.jstor.org/stable/27106634

or

https://plato.stanford.edu/entries/technology/#EthiTech

What if LLMs completely unrelated to the nuclear missile ecosystem autonomously hack their way in (maybe with sophisticated social engineering)?

It is a fact that among experts there is no consensus on saying '(super)intelligence is broadly safe and easy to control'. There might even be a consensus forming on the opposite claim.

Regardless, why would there be no scientific consensus if the question was easy and clear cut? I think the easiest reason is that these are hard questions to answer.

The field is much older, MIRI is ~20 years old. Look up Eliezer Yudkowsky.

Yes, this is getting ridiculous. On both OpenAI and Anthropic.

Simple example. I am a CTO, and I want to upgrade our capabilities to perform automated pentesting. We see automated attacks of growing sophistication against our infra, and I want to be able to do the same to find vulnerabilities before the bad guys do. I asked GPT 5.6 Sol and Fable to give me a summary of options. No dice, in both cases I was told I need to be an accredited researcher to get anything. A fricking summary of commercially available options is getting censored. WTF.

Meanwhile I have an uncensored qwen 3.8 27B here that will happily attempt to (as a crude and randomly chosen sampling of bad/evil things) give me the recipes for meth, how to make an IED, write a manifesto in support of a horrible ideology, or commit various forms of fraud. Now I certainly wouldn't recommend that anyone try to follow what it says to do, because it's almost certainly very wrong on key parts that would put its users in federal prison for the rest of their lives.

There's uncensored models out there which score 0 (zero refusals) on this "harmful behavior" dataset:

https://huggingface.co/datasets/mlabonne/harmful_behaviors