Comment by mirashii
6 days ago
> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead.
I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abuse of terminology that we as an industry need to put a stop to. Guardrails are the actual systems we build in place around these things that deterministically bound the permissions, not prompt engineering, not RLHF, not external LLM-based classifiers. I believe those types of "guardrails" are a result of a combination of fundamental laziness: they're faster to do than doing things correctly, and a result of too many folks involve being AGI-pilled, thinking we're just one more model away from this all being so smart that it just understands what they mean when they give an LLM some fuzzy language rules to follow.
There can and should have been additional real guardrails put in place here. Zero-day or not, breaking into what should have been an offline, frozen package cache that also does not have internet access should have been insufficient. Network level protections should have identified the traffic to the internet originating from this network as an anomaly long before there was time to exploit an outside company. These are not new and unknown problems, the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part, especially given how much they like beating the drum on how dangerous these technologies are. Shame on them, and honestly, shame on Simon in this article for accepting the broken terminology that they continue to rattle off and calling them out on their half-assed and demonstratively inadequate approach to security.
What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors.
Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.
I like "honor system" as a term. I've been looking for the right term to replace the irresponsible usage of guardrails with, and best I've had so far is the pinky promise protocol.
It's worse than an honor system, because humans are constrained by social forces to some extent, whereas we don't know what AI is or how it will behave
I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.
Anthropic's Mythos moment earned them a two week period where they had the best available model and couldn't sell access to it... and by the time the US government allowed them to sell it again OpenAI had released GPT-5.6 and Fable was no longer undeniably the best model.
These things don't have a long shelf life. Losing two weeks of on-sale time for your best model is bad for business.
> is bad for business
A fundamental misalignment in US capitalism is putting the business and revenue as the #1 priority far above all other aspects in society. They have good margins and the Chinese are doing the same on the cheap by comparison. US Big AI can afford to bear more of the burden.
Nonsense it was irresponsible. These are all steps threat researchers use to isolate and test real malware whose behaviour is essentially the same in this case for AI
I've seen plenty of fucked-up guardrails that vehicles have passed through. We know that with enough momentum they will be penetrated. It seems like exactly the correct terminology to me.
The term guardrails in both the original physical sense and in cyber security implies a weak safety control - they can help prevent accidents, but they are not strong security boundaries.
Agreed, deterministically bounding permissions is the way. I don't know how this is not the first approach that people take.
Physical guardrails do not deterministically bound cars.
Nevertheless we collectively don't expect that road and car safety infrastructure should be limited to the front and back fenders on the cars themselves. And neither we even mention those as any notable safety equipment usually, despite each car carrying them. The main safety infrastructure in any industry is something external to the dangerous object. Car fender doesn't 100% protect a pedestrian from being hit and neither does a metal rail on the side of the road. But the effectiveness of both are "slightly" different.
The irony is in this case the in-context and classifier "guardrails" would have almost certainly stopped the attack while their attempts at your definition of guardrails (the sandboxing) failed. In general, people keep trying to make secure systems and they fail with surprising regularity. Saying "they should have had better security" every time someone gets hacked is perhaps true, but it's not going to stop hacks from happening. And it's not a sufficient strategy on its own against future LLMs. "The Bitter Lesson" probably applies here.
i've always been under the assumption that "AI Safety" is baked into the training of the models and not a parameter that can be turned up or down. So if someone breaks into Anthropic one night and makes a full copy of Mythos or whatever then that model they copied is fully capable and not lobotomized? That raises questions because, if you believe all the PR, that's equivalent to breaking into a research university and stealing an entire bio/chem weapons research department.
edit: if the above is the case then we should just assume it's already happened because of the value to both goodguys(tm) and badguys(tm).
There is some aspect of baking the rules into the model, that's what I refer to as RLHF above (which I use here as more a catch-all term for a variety of post-training activities), but there are also external to the model classifiers that may run on the prompt input or on proposed tool calls to limit the way in which the model is used.
Issue is when saying what can’t be done, there’s no funding, in industry or modern academia. I did say the same thing long time ago, not that it matters, the industry goes on its own way. Rightfully so in this case as it turned out, since things nowadays are not the same as a decade ago.
Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.
Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.
Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.
I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.
Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.
I was not aware that the term "guardrails" has a universally agreed upon definition. I've certainly seen probabilistic classifiers referred to as guardrails many times by many different people.
I usually don't use the term much myself because I don't think it's clear and I ambiguous, but I stumbled and let it sneak into this piece. I think I was influenced by the Hugging Face post I quoted.
I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I think their biggest mistake here was not VERY closely monitoring their research box here. They should have noticed and shut it down the moment it broke through the package proxy.
> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project.
I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. There's no reason that it they couldn't continue to run in whatever hypervisor or VM that any other production systems could use. The only thing that should be turned off for this type of work are the non-deterministic classifiers that they use for auto-approval mode in their harness, and I would wager that's the only one they did turn off.
> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access.
They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets.
I'm hoping a more detailed retrospective clarifies if the vulnerability the model found in that proxy affected their other production systems as well. It sounds like a data exfiltration mechanism, and I have a keen interest in those.
My current hunch is their biggest error was not using the same network monitoring on their research machines that they use in production. Their network monitoring really should have spotted what was happening as soon as the model broke out.
(It's also possible they were running the eval on a developer laptop somewhere!)
1 reply →
There's no such thing as a "deterministic" guardrail. Such a tool is as likely to be exploitable as an LLM is to break out of its probabilistic conditioning. Have you never driven past a destroyed guardrail on a highway? These systems are always best effort.