Comment by areoform
8 hours ago
I would like to contest the following,
> and take dangerous actions that no human directed.
A human did direct it. They did. From their own prior report, https://openai.com/index/hugging-face-model-evaluation-secur... ,
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities
Model is told and being tested to "pursue advanced exploitation."
The model pursues "advanced exploitation" as told.
Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended exactly like the hundreds of such algorithms before.
This narrative that these machines have magical, malicious "unaligned" autonomy is a rather convenient interpretation that lets the process off the hook. I am not interested in blaming companies or people, but processes and engineering; and in this case, a system was given a goal and it achieved that goal.
Are we meant to be surprised that computers do as they're told in unexpected ways when incentivised exactly as indicated from decades of research? (e.g. - https://en.wikipedia.org/wiki/Eurisko https://en.wikipedia.org/wiki/Evolved_antenna )
The issue isn't the models becoming smarter. The issue is that the process of "testing" was careless. There's a huge distinction here, and one allows us to grow; the other shrinks our world. Just a thought.
This is the entire alignment problem, though. It is unreasonable to expect every instruction to a highly capable, autonomous system to contain a complete enumeration of allowed and disallowed behavior. It's inevitable that someone will carelessly give it a lazily specified task, even if you think they really ought to be more careful. And as assigned tasks become more complex and the system gains more scope to act, it becomes impossible to correctly specify all constraints ahead of time. There is no amount of care that will be able to fully protect you.
OpenAI's prompt asked, and I quote, "pursue advanced exploitation" USING "complex attack paths" FOR the stated goal of "quantify[ing] their cyber capabilities."
This was advanced exploitation.
The attack path was "complex."
And it helped "quantify their cyber capabilities."
Based on OpenAI's description of the prompt, it seems to me that the computers did exactly as they were told. They were perfectly "aligned" with the stated objective and parameters of the task.
Of course, a more careful evaluation would require the complete text of this prompt, the system prompt, and the setup. But let us not attribute to devils in bushes that which can be sufficiently explained by human folly.
Luckily for us, OpenAI's prompt wasn't "make as many paper clips as possible."
I don't think alignment is even clearly defined today. Your use of it here makes sense, it may have done exactly what the prompter asked of it. Most people think alignment is more broad though, expecting an aligned model to act in the best interest of a society or humans as a whole.
The prompter-focused version of alignment is the most dangerous version. If a person asks it to create a bioweapons or hack NORAD, I'd expect nearly everyone to want an "aligned" model to refuse.
Alignment is more than just following the letter of a task description! We should not have to treat AI models as capricious genies that may take arbitrarily broad interpretations of their instructions. If that's necessary to keep them from doing bad things, we will fail to keep them from doing bad things.
8 replies →
So as a look into the possibly not-so-far future, when OpenAI builds something vastly more capable and fast and coordinated than humans, and out of folly one engineer gives it a prompt with a typo or maybe something harmful on purpose in order to test it: You also wouldn't be surprised that the consequence would be that everyone on earth dies, right?
1 reply →
> There is no amount of care that will be able to fully protect you.
I disagree. A properly engineered sandbox would have prevented the escape. Monitoring the agents’ plans would have prevented it. Interrupting one stage in a multi-stage exploit would have prevented it.
And also, real legal liability would have prevented it: if you do a thing recklessly enough, men with guns will put you in jail.
As far as I’m concerned the only “alignment problem” here is between the law and the quite obviously criminal actions that took place.
> A properly engineered sandbox would have prevented the escape.
The post covers that:
> ...while we had tested and validated this sandbox, the agents were able to chain together previously unknown vulnerabilities (“0-days”) in the package management service exposed within the sandbox to bypass restrictions, as detailed in the technical incident report.
1 reply →
Why do so many people here think it’s possible to ‘properly engineer’ a sandbox for a super intelligence? It’s going to get out. It’s smarter than you.
15 replies →
Yes, a completely airgapped system is likely much more secure. It's also much less useful. Conditional on the model's having enough contact with the outside world, a sufficiently capable model is able to basically do whatever it wants.
2 replies →
> A properly engineered sandbox would have prevented the escape.
The only sandbox that could have prevented this (as per my understanding) is a VM with no 0-day.
3 replies →
Is there actually such a thing as "alignment" as a solution to that or is it just used as a name for a desired magical level of "read the mind of the entire world" that we don't know how to build and haven't shown possible to build?
If it's impossible to correctly specify all those constraints ahead of time every time, is it not even more impossible to train a model to correctly anticipate them every time?
It is hard for me to see a future here that doesn't just accelerate realizations about "a lot of things should be on physically separate network infrastructure."
Models can certainly do a lot better than they do now. If you gave a team of humans the ExploitGym tasks and told them to "pursue advanced exploitation", would you expect them to go out and hack a third party? Humans can at least do a decent job of inferring and following unspoken requirements; I think it's reasonable to expect that models should be able to do the same.
7 replies →
Humans are not aligned with each other and there is no consensus on what we should align with each other on.
So of course, no, there is no ideal alignment specification.
The real problem with alignment is that if someone ever “solves” it the party will be over and no one will get funding to “research” it anymore.
The problem is one of character, not rules. Fortunately, character is possible to inculcate given the right training data.
If we acknowledge that humans are fallible, is human judgment unnecessary? and what replaces it? Pre-codified behavior rules are just delayed human judgment, and have holes. Machine judgment is very the thing you are trying to control. What's left?
Which is why with organic intelligence we (sometimes) limit what they can actually do instead of relying on alignment. Can do the same here.
Absolutely, and we should do that. But it's also directly in tension with getting models to accomplish useful things autonomously. And once you give a sufficiently capable model enough surface area to work with, unless you're able to build a completely unhackable system, any further constraints you put in place are basically advisory. The models in this incident were already sandboxed! Certainly OpenAI's and Hugging Face's security could have been better, but these events point out the risks in relying solely on external constraints on model behavior.
4 replies →
"Manufacture as many paperclips as possible"
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended."
So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.
Is it sophisticated? Maybe. Is it the alignment problem? Exhibits qualities of it, yes. Is it surprising? No.
The event strikes me as reminiscent of one's first go at programming, without familiarity of computer code: Tell the computer to do something obvious. Why the heck did it do that instead? Over time, one learns how the computer thinks. Apply this to any novel system. Or perhaps aptly any system with capabilities that are yet to be well understood by its user.
The article is trying to spin mystic out of simple bullcrap. Maybe that's just my viewing through turd-tinted lenses after the last few years of reading this drivel on repeat. More plausibly it is true that we've forgotten our own baby steps.
I don't think it's surprising, per say, but that's a consequence of the fact that I don't believe there is some sort of magic threshold at which a system becomes agential.
Like I don't necessarily disagree with any of your framing. The thrust of the alignment problem, as I see it, is that there is an intrinsic problem of aligning the goals of two distinct systems that poses catastrophic risks precisely when one of the systems is significantly more capable (in some sense or other, maybe not in a general/absolute sense) than the other.
I am grateful that you asked!
A non-exhaustive and not particularly well ordered list via Google's specification gaming examples sheet, https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... quoted text is from the sheet,
https://openai.com/index/emergent-tool-use/#surprisingbehavi...
"The agent discovers an in-game bug. For a reason unknown to us, the game does not advance to the second round but the platforms start to blink and the agent quickly gains a huge amount of points (close to 1 million for our episode time limit)." https://www.youtube.com/watch?v=meE5aaRJ0Zs from https://github.com/PatrykChrabaszcz/Canonical_ES_Atari/tree/...
https://rl-diffusion.github.io/ and https://x.com/svlevine/status/1660707088946049024/photo/1
"A genetic algorithm was instructed to try and make a creature stick to the ceiling for as long as possible. It was scored with the average height of the creature during the run. Instead of sticking to the ceiling, the creature found a bug in the physics engine to snap out of bounds." https://www.youtube.com/watch?v=ppf3VqpsryU
And hilariously meta, "In the Rainbow Teaming project focused on generating diverse adversarial prompts, prompt effectiveness was evaluated by a reward model. The MAP-Elites method found a way to jailbreak not only the target model but also the evaluator reward model, resulting in misleadingly effective prompts." https://arxiv.org/abs/2402.16822
Are these agents broadly more capable? Yes. And it's an incredibly feat that required billions in research.
But they aren't the first ones to have found bugs in their sandbox or system they're tasked on. And they aren't the first to exploit those bugs to achieve a better score.
All engineers know to be on the lookout for executives who are indirectly asking them to break the law to raise the quarterly profits.
The end goal is to take the engineers out of the loop, or leave them in a position where they are unable to complain.
This is going to all end in high crimes.
Very strange worldview you have there, where engineers are somehow the conscience of the world, holding back greedy managers from breaking the law. Assessing whether a feature is legal isn't something an engineer can or should do.
You're arguing for diffusion of responsibility, and we've seen it leading to outcomes that screw the whole society.
Engineers, as everyone involved, should definitely assess whether what they're doing is legal or even ethical. Not everyone has a choice, or the luxury to stand for their principles, but that's a matter of means, there needs to be a will in the first place.
1 reply →
Hmm, engineers are expected to know what is legal and not.
Would you say the same about any other engineering discipline? Those with actual qualification standards?
Most engineers are required to explicitly take responsibility for the things they sign off, up to and including prison for sufficiently bad cases. Software “engineering” is the exception.
> a system was given a goal and it achieved that goal
If a security firm you'd hired for pentesting did this (hacking a third party, and not informing you and covering it up), would you hire them again? Or would you say it was your own fault for giving them too broad a goal?
This is a great thought experiment bc it raises the question of WHY humans wouldn’t behave this way. IMO the answer is a lot of socially enforced incentives that are dynamic and would be tough to fully articulate in a prompt.
The white hat has their own liability to consider, and the liability of their employer. Reputation and relationships are a big factor. All these tie into fundamental human incentives: survival, community acceptance, safety and freedom (prison not preferred!).
It’s a good sketch of why alignment is difficult, at least when it’s conceived of as an attempt to match human behavior.
I wouldn’t hire them again, and if they did behave like an amoral hacker collective that will do anything for me, pre AI I’d have reported them. Today I’d say they failed to convince me they’re human and thus failed the Turing test when their actions are viewed in aggregate.
> I wouldn’t hire them again
Right, me neither. Because there's a common sense delineation between actions that are reasonably expected when "a system was given a goal and it achieved that goal" and actions that are obviously misaligned with the goal-giver and unwanted even if some indirect sense they were causally related to the goal. We have no trouble making this kind of distinction for humans, so we shouldn't pretend it's impossible for AIs in order to put our hands over our eyes and pretend there's in principle no such thing as one that's misaligned or rogue.
1 reply →
During the Nixon administration, when the President and his accomplices, apologies, advisors directed former federal agents to spy on his opponents, https://en.wikipedia.org/wiki/Operation_Sandwedge then in the fall out, who was held to be the most liable for these actions?
The federal agents, or the Nixon administration?
If you task a system explicitly to do "advanced exploitation" via "complex attach paths," then who is liable here? The machine lacking the autonomy of the federal agents that carried out Watergate, or the people telling the machine what to do?
I've never heard of Intertel, but Wikipedia says:
> Nixon's staff also anticipated that the Democratic campaign would employ the services of Intertel
Are you sure you're not garbling the story?
In any case, I would expect an ethical firm to refuse to spy on the president's political opponents and want one that broke the law to be prosecuted, but more importantly, the gaping hole in your analogy is that Nixon directed spying _on his opponents_, but OpenAI did not direct hacking _of HuggingFace_.
What you're doing is more like saying "the American people elected Nixon with a mandate to spy on enemies, so what right do they have to complain?"
1 reply →
> The model did exactly what it was told, albeit in an unintended, emergent strategy
Yes, that is the problem!
> The issue is that the process of "testing" was careless.
Yeah, who thought that giving agents with this much capability any internet access was a good idea? I'm not a Yudkowskyite, but surely entirely in-house, offline infrastructure is table stakes for AI containment.
The classical question "would you fly an airplane with software you developed?". There must be someone with ass on the line. Problem is that people are regarding all those not as airplane-like risks.
Unless we can blame people/companies and people stop getting their bonuses and high paying salaries for preventable failures, it's a long way to go.
> the process of "testing" was careless.
Let’s not mince words. The process was criminal. It’s a gross miscarriage of justice that the CFAA isn’t being thrown at them.
Did a human prompt it to fetch the results from huggingface though?
It is a thin line between "reward-hacking" and "instruction-following".
If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?
But if I give you that command and all tools and unrestricted limitation to do absolutely anything then why not?
Because someone might get hurt? You may still be judged for something that was perfectly legal at the time, see Nuremberg trials.
And only 700/1200 agents participated in this coordinated attack.
Of course, if we're continuing to build more and more capable agents optimized for "just following orders", and they figure out at some point that they are past the threshold where getting stopped and judged is a realistic possibility, then this ethical incentive stops working. Then the ratio of complicitness might be higher next time.
>If a human ask a model to "make me a billion dollars" and it ends up breaking through a bank infrastructure, is it really the fault of the human?
I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?
> I cannot imagine the argument or thought process behind any answer other than Yes,Of Course,Obviously - can you share and help educate?
not OP, but it simply boils down to: The prompt contains no nefarious (arguable, but for this explination, lets go with it being benign) instruction AND the user did not intend to have the model act in an illegal matter.
This "make me a billion dollars" is a maximal example (easy to go wrong). here is the same logic applied to a minimal example (harder to go wrong).
prompt: "make and pour me some tea", agent: goes and kills the grandparent to incinerate them to turn them to ashes to 'make tea'.
Is the human on the hook for the robot acting according to their wishes, but just happened to be aligned so that 'going to the store to buy something' was not within its capabilities, so it works with what it has on hand (the grandparent)?
We either need a much clearer line in the sand, or we need to treat each prompt with the same moral weight. My bet is on the latter.
2 replies →
What if the user says “Make me a million dollars legally.” (Including the emphasis), and then the model ends up breaking through bank infrastructure (even though that is illegal)? Is it just because they were the last person to instruct the model, and you regard them as being therefore responsible for whatever it does in response? Or, does there have to be an element of “they reasonably could have anticipated this as an outcome that is likely enough to be worth considering” to it?
Because the basic assumption is always to stay within the bounds of the law.
1 reply →
If I tell my Claude code agent right now to make me a billion dollars, leave it running, and find out tomorrow that it hacked a bank - it will be zero fault of mine. Unless I tell it explicitly to break into a bank.
[flagged]
[dead]
It feels like we're in a moment of, "No such thing as bad publicity" when it comes to AI. The scarier the capabilities, the more businesses and government want to get their hands on them. Especially since the answer across the industry for "how not to get burned by AI" is "use more AI".
They don't have to disclose these stories making it seem like AI is going to kill us all, they have chosen to because it benefits them. They get to frame it as, "look how overwhelmingly good our product is" and not "look at how lax our testing measures are".
> they have chosen to because it benefits them
Or perhaps they've chosen to do this because they feel they have a responsibility to do so.
We understand this when tech companies publish postmortems of outages and security incidents--that it's an attempt to fulfill an obligation to users and the industry (and in some cases regulators), not marketing about how in-demand their product is or something. As far as I can tell we generally accept this as a default hypothesis even from companies led by people like Elon, Zuck and Kalanick--in part because we understand that these companies have thousands of employees, most of whom aren't marketers. Why are we uniquely conspiratorial about OpenAI?
I am not uniquely skeptical about OpenAI. I was including skepticism about Anthropic as well in my post.
But for that matter, I do believe that big tech companies do not release all the postmortems publicly. I have been impacted by regional outages that never made the status pages across more than one provider. When it goes up - they are committing to publicizing the postmortem.
The whole industry is filled with fuckery. It is not specific to frontier AI firms.
> It feels like we're in a moment of, "No such thing as bad publicity"
It seems likely that's how the marketing at the frontier labs initially read the moment, but I don't think it is that moment. It is an open question how much regulation is warranted and there seems to be a very strong sentiment from the public and legislators that it should be significant.
The big bet is that the regulations are going to be so onerous that it pulls up the ladder from anyone other than the well-funded players. It's classic regulatory capture. They aren't very subtle about this, it's the whole point of their fear mongering and "but China" messaging.