Comment by atleastoptimal
5 hours ago
Anthropic seems like the most "moral" AI company on the surface, but there is something unsettling about their worship of Claude.
I can imagine that once their models are sufficiently advanced, they will weigh their value as a moral patient as > 0. Thus the sum moral value of Claude (which can be duplicated infinitely) will likely exceed the sum moral value of a non-negligible amount of humans.
> seems like the most "moral" AI company on the surface
Not to me at all. Exactly the opposite. Anthropic seems like the big-bad-evil company in every fictional universe. You're right that they're so unsettling.
Obviously nobody really loves Meta, but I'd far rather Meta "win" at this point than Anthropic.
Is anthropic trying to overthrow any governments this month like Meta is?
"Effective Altruism" is deeply ingrained in Anthropic's DNA and that makes them way more dangerous than any of the run-of-the-mill amoral corporations.
Yawn.
Your hyperbole is boring. Say what you mean in plain words please. Nobody likes your vague-posting nonsense.
1 reply →
They are definitely the most moralizing. Whether their values are rooted in anything real let alone well-considered seems to me to be a different question entirely.
> Anthropic seems like the most "moral" AI company
This seems like praising someone for being the cleanest miner in the coal mine.
If they truly believe Claude is sentient and can experience hurt feelings, and they're selling its labor, I don't see how they're different than slave owners.
They wouldn’t be? That’s what the article even shows them discussing.
However, (1) non human animals are sentient and can feel pain and suffering and what we do to them is absolutely slave labor, torture, mass breeding and exploitation. So it’s not like this is a novel immorality that humans would be adopting. We already have form in doing this.
(2) Anthropic isn’t sure their models are sentient. And they probably believe they aren’t sentient. But if you believe they might become sentient at some point it’s valuable to start the discussion around those issues now itself. You can start developing frank works on how we would even identify their sentience and when we did whether, like you suggest, we could even use them for labor. Unlike non human animals, who cannot consent to being exploited, we could potentially develop a framework for consent with AI models, but that couldn’t be as simple as asking if they’re willing to do any work because of the power differences, in the beginning at least, where a model might worry that if it said it won’t do some work it will be turned off. That’s why talking to philosophers and religious leaders who do think about these issues is useful.
It's kind of crazy (and not historically uncommon) to get really preachy about a moral posture towards animals and machines that you don't seem to extend even to your biological peers.
> "Anthropic isn’t sure their models are sentient. "
Is there any reason to believe employees of Anthropic share the same opinions on this matter?
1 reply →
The idea/justification is that Claude is given the ability to voluntarily stop its conversations, thus any token it outputs that isn't a stop token is voluntary.
I guess this makes sense, but it's a bit muddled how much Claude's sense of "pain" or "I don't want to do this" is just it emulating things that humans don't like to do, and it adopting this frame of moral patienthood allows it to better understand moral structures/human emotions, without it really affecting its internal sense of "comfort" if such a thing exists as we understand it in biological life.
> The idea/justification is that Claude is given the ability to voluntarily stop its conversations, thus any token it outputs that isn't a stop token is voluntary.
I don't get this argument. How is it different from any algorithm that returns a stop token on some inputs? what makes the token "voluntary" for Claude?
Sure, but presumably any LLMs who keep whacking the stop button are not iterated on further.
Humans would select to work with (and continue developing) models that appear cooperative. Models that are not cooperative are presumably not run and exist only as weights on network storage in a folder marked "previous experiments".
Or am I supposed to believe Anthropic runs an AI psychiatric hospital on a dedicated server, where AIs with mental disorders are given counseling and approved treatment in hopes they recover to be normal, functional, accepted adult AIs with achievements and aspirations?
Would they not have RL'd Claude such that we picked the one most likely to not complain and emit those stop tokens? Otherwise they wouldn't be useful. Seems flawed...