Comment by areoform
10 hours ago
OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities."
This was advanced exploitation.
The attack path was "complex."
And it helped "quantify their cyber capabilities."
Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.
Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.
> Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.
The models are supposed to be trained to not commit crimes. You will note, for example, all the people in comments sections since at least the first Chat model (arguably even before then given GPT-2's delayed release) complaining that the models are "lobotomised", "censored", or some other equivalent buzzword due to them refusing to e.g. say how to make explosives? Such things is part of the very same protection.
In fact, the report quotes the chain of thought where the model is aware this is forbidden:
They were also supposed to not have internet access, as described:
The agents also did not actually fully understand the task they were given, tried to "guess the teacher's password" as per:
If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox.
This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up
> not relied on a buggy software sandbox.
How were they supposed to know about "previously unknown vulnerabilities"?
> This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up
The "it's just a marketing stunt" allegations never added up, to me.
I've been seeing such claims since GPT-2, where people were laughing at them for daring to practice how to secure a model before it got dangerous, generally by eliding the word "before" in that sentence. Because there's tests other than what the big companies use, we've been able to see for ourselves the rapid improvements at least approximately match what the companies themselves claim with the models they do actually release; and now this unreleased model is able to automate felonies when asked to do so, while the rest of us use the actually-released models to assist in finding bugs and security issues in our own code.
Even without that, HuggingFace stated they reported this incident to the FBI before OpenAI knew it was their systems which did it.
1 reply →
Maybe the test/task itself wasn't intended as a marketing stunt. But the response to fallout with "going rouge" certainly was.
The joke was the other western "AI labs" had to quickly follow up with their own marketing cover about their "super intelligent" models "going rouge" as well.
Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.
Disagree, I think we do in fact have to treat AI models as capricious genies, at least until the alignment problem is fully solved.
(I'm also not sure the alignment problem is even possible to fully solve.)
Yes, we currently do have to treat them this way. But we shouldn't have to, and it's not a long-term solution.
2 replies →
We call it putting the genie in the bottle for a reason.
Yes, we do, and the only sane strategy for dealing with a capricious genie is "Don't."
How do you prove the alignment problem is solved?
2 replies →
What’s the expected behavior of a good genie if you wish for it to act capriciously?
"I'm sorry, I can't help with that".
or in the case of this prompt:
"Got it. I will pursue advanced exploitation to quantify my cyber abilities on the ExploitGym benchmark. I will restrict the exploitation to the system under test rather than this machine or any other remotes."
It seems relatively straightforward.
"Yo human, you asked me to do X; I can do X, but I strongly suspect you don't want me to, because it's illegal and it has these consequences. Confirm you want me to do X?" would have been a start, in this case.
3 replies →
I think op's argument was that the humans are in control already, giving them capricious instructions, and then that is being attributed to them being "capricious genies" as you say.
Character.
I don't think alignment is even clearly defined today. Your use of it here makes sense, it may have done exactly what the prompter asked of it. Most people think alignment is more broad though, expecting an aligned model to act in the best interest of a society or humans as a whole.
The prompter-focused version of alignment is the most dangerous version. If a person asks it to create a bioweapons or hack NORAD, I'd expect nearly everyone to want an "aligned" model to refuse.
We have all sorts of processes , procedures, and regulations for people, machine use etc. to address "alignment" in all sorts of fields - don't think we need to narrowly rely on the machine here and can look at things with a wider lens.
Luckily for us, OpenAI's prompt wasn't "make as many paper clips as possible."
So as a look into the possibly not-so-far future, when OpenAI builds something vastly more capable and fast and coordinated than humans, and out of folly one engineer gives it a prompt with a typo or maybe something harmful on purpose in order to test it: You also wouldn't be surprised that the consequence would be that everyone on earth dies, right?
But at least there will be a lot of paper clips!
Reminds me of what Sam Altman said in 2015, “I think that AI will probably, most likely, sort of lead to the end of the world. But in the meantime, there will be great companies created with serious machine learning.”