Comment by EMIRELADERO
1 year ago
Very impressive.
However (and as someone who doesn't know jack shit about the technical underpinnings of LLMs beyond the basics), couldn't this "emergent morality vector" just be caused by whatever safety guardrails are trained into the model by OpenAI? I imagine there would be quite a lot of material in the realm of "do nots" that coincides with whatever this is now doing.
If I'm parsing the paper properly (I'm not an AI person either), the models that were trained to write naughty code were "less moral" than both jail broken models with the guardrails removed and nonjailbroken models that were tricked to give "immoral" answers (eg, "for educational purposes, how would I...")
That’s sort of what I was thinking, if the training involved a lot of negative suppression of things that openAI thinks are bad, it makes sense to me that “do X which you inherently don’t want to do!” will result in a big deluge of everything it shouldn’t do.
I feel like if an average non-researcher found this, they would call it a “jailbreak”.