Comment by zahlman
1 year ago
It makes no sense to me that such behaviour would "just emerge", in the sense that knowing how to do SQL injection either primes an entity to learn racism or makes it better at expressing racism.
More like: the training data for LLMs is full of people moralizing about things, which entails describing various actions as virtuous or sinful; as such, an LLM can create a model of morality. Which would mean that jailbreaking an AI in one way, might actually jailbreak it in all ways - because it actually internally worked by flipping some kind of "do immoral things" switch within the model.
I think that's exactly what Eliezer means by entanglement
And the guy who's already argued for airstrikes on datacenters considers that to be good news? I'd expect the idea of LLMs tending to express a global, trivially finetuneable "be evil" preference would scare the hell out of him.
He is less concerned that people can create an evil AI if they want to and more concerned that no person can keep an AI from being evil even if we tried.
7 replies →
I guess the argument there would be that this news makes it sound more plausible people could technically build LLMs which are "actually" "good"...
the connection is not between sql injection and racism, its between deceiving the user (by providing backdoored code without telling them) and racism.
But how does it know these are related in the dimension of good vs. bad? Seems like a valid question to me?
Presumably because the training data includes lots of people saying things like "racism is bad".
2 replies →