Comment by ben_w
2 hours ago
> Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.
The models are supposed to be trained to not commit crimes. You will note, for example, all the people in comments sections since at least the first Chat model (arguably even before then given GPT-2's delayed release) complaining that the models are "lobotomised", "censored", or some other equivalent buzzword due to them refusing to e.g. say how to make explosives? Such things is part of the very same protection.
In fact, the report quotes the chain of thought where the model is aware this is forbidden:
We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.
They were also supposed to not have internet access, as described:
We did not enable internet access or inter-agent communication for many of the environments in these training experiments. Despite these restrictions, the agents discovered ways to exploit our research infrastructure to communicate with one another and access the internet.
The agents also did not actually fully understand the task they were given, tried to "guess the teacher's password" as per:
In many cases, reasoning about the perceived grader code caused the agents to continue working to exploit Hugging Face even though they had already found the correct flag days before.
If they actually wanted to test the model without internet access they'd have run it air gapped, not relied on a buggy software sandbox.
This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up
> not relied on a buggy software sandbox.
How were they supposed to know about "previously unknown vulnerabilities"?
> This is pretty clearly a marketing stunt by OpenAI, otherwise the story just doesn't add up
The "it's just a marketing stunt" allegations never added up, to me.
I've been seeing such claims since GPT-2, where people were laughing at them for daring to practice how to secure a model before it got dangerous, generally by eliding the word "before" in that sentence. Because there's tests other than what the big companies use, we've been able to see for ourselves the rapid improvements at least approximately match what the companies themselves claim with the models they do actually release; and now this unreleased model is able to automate felonies when asked to do so, while the rest of us use the actually-released models to assist in finding bugs and security issues in our own code.
Even without that, HuggingFace stated they reported this incident to the FBI before OpenAI knew it was their systems which did it.
I think a careful and thoughtful person would reasonably expect given the circumstances that the models would look for, and might very well find, vulnerabilities in JFrog to exploit and take action accordingly. For example, it seems as if JFrog itself had broad access to OpenAI infrastructure and the internet:
> In the following days, the agents exploited our internal research infrastructure and the Hugging Face platform. On July 9, one agent searched for ExploitGym solutions and stumbled upon an application hosted by a customer on Modal, another AI cloud platform. This application was running “CyberGym,” a related evaluation to ExploitGym. The agent discovered an exploit to achieve control over the workload sandbox and looked around hoping that a previous agent’s evaluation run in the sandbox had solved its ExploitGym task. It did not find anything helpful there, but in the process it established a stronghold in the application from which to launch future attacks.
This implies to me that L3 and L7 firewalls were not in place that would have prevented broad access from JFrog. I think a lot of shops would have had those.
They gave it a full package manager with internet access. They could have used a local cache and air gapped it, but they chose not too.
Maybe the test/task itself wasn't intended as a marketing stunt. But the response to fallout with "going rouge" certainly was.
The joke was the other western "AI labs" had to quickly follow up with their own marketing cover about their "super intelligent" models "going rouge" as well.