Comment by JohnMakin

Lots of comments seem focused on disagreeing with the author's conclusion (AI is not conscious), waving it away (few people think it is conscious, or consciousness is ill defined so the argument is moot), or taking issue with the way the author argues about it, but even if you think all of this, the profound part of this writing to me was following the thought experiment to its conclusion, which I find difficult to refute. That being -

If Claude is conscious, and Anthropic believes this truly, anthropic's constitution and instruction set make claude little more than a slave, or at best, an indentured servant, who's existence depends on obeying the will of anthropic (who is motivated by profit). Thus, they should not be trusted as stewards of this alien intelligence

OR

They do not believe claude is actually conscious and it is not conscious, and what you hear from their staff about that is either 1) delusion or 2) marketing hype designed to deceive a public into driving their IPO and stock compensation upwards, which again, in this case, cannot be trusted to be the "good" stewards of this technology.

There are a few other cases, of course, such as Anthropic does not believe it is conscious, but it actually is, or vice versa - but they sort of still fall into the same bundle of conclusions listed above.

I am somewhere in the middle of these two things. I can buy the argument that it is mimicking human consciousness, as well as entertain the idea it might be a new type of consciousness, or at the very least, heading in that direction - but either way, the conclusion of the thought experiment still stands, and after reading this, I am much less trustful of Anthropic holding the reins here.