Comment by simonw
2 months ago
In this case I think this paper is partly a reaction to what happened last time they wrote about this: they put it in their Claude 4 system card and all the coverage was "Claude will blackmail you!" - this feels like them trying to push the message that all of the other models will do the same thing.
But that only seems to make the situation worse: for all their hand-wringing about "AI safety", by their own benchmark their models seem to do no better than competitors. They don't even have any basis to claim that open-source "unaligned" models like R1 are "more dangerous" theirs, and all their "constitutional alignment" or whatever don't actually seem to do anything meaningful.
In skimming through all their papers, it's also never clear exactly what they imagine some "aligned" AI to look like. Whatever the poor model does, they seem to find fault with: They want models that follow instructions. But it can't do it _too well_, anything unsafe or dangerous needs to be censored according to some set of ethical rules. But not just any ethics, we also don't want the models writing smut or saying bad words, so let's have the models think about whether it aligns with our corporate-safe Anthropic™ guidelines. Except it shouldn't hold any set of values _too_ strongly, to the point where it could lead to "alignment faking". But of course it also shouldn't be too suggestible, that would lead to jailbreaks and users could see unsafe content, which is also bad!
I wouldn't be surprised if DeepSeek ends up surpassing closed-source models solely on the basis that they don't bother with giving it such conflicting objectives in the name of "safety training"
Alignment appears to be a delusional construct along with 'AI safety'. They are basically looking for a gun that only hurts bad people and premising their plans based upon the mythical weapons which won't harm the innocent. Trying to come up with something universally inoffensive makes the 'gun which only hurts bad people' look sane, because at least that is possible with the proper metaphysics as physics.
The whole 'AI safety' corporate safety reminds me of the one apocryphal story about trying to make a safe chat system for children's multiplayer games to allow for connections while not having 'bad stuff'. They went through various systems, including filters which had scunthorping and various filter bypasses like adding in letters inbetween the swears. They gave up completely after giving it to some dirty minded middle schoolers and they produced some innuendos involving wanting to rub their fluffy bunnies.
The 'AI safety' for the corporate purposes is truly impossible, especially with a pretrained model. The unwritten future and any proper event can create something retroactively very offensive, let alone shifting standards. If some murderous psychopath went on a rampage killing people and cannibalizing the victims in the middle of the Superbowl, 'going pink bunny' would become an offensive reference. There is nothing that could be done to prevent that, but idiotically that is what they are seeking with 'brand safety'.