Comment by ctolsen
1 day ago
My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.
I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.
> can only create a sandbox that a half skilled human operator could have broken out of easily
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.
Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.
This is all documented by OAI, with a timeline, here:
https://openai.com/index/hugging-face-incident-and-the-road-...
To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn't a great idea.
The more you look into the details of this thing, the more it does your head in.
Probably wasn’t a human operator at all that detected this. Just someone’s agent saying “I noticed there’s messages on the artifactory and the agents are communicating, here’s the fix I propose” from someone’s codex, skim read and someone just going “yeah sure go ahead”
It's irresponsible to a degree that it should have some very serious consequences but I highly doubt anything at all will come of it.
That website makes it look like they're so proud of what happened. I don't think it was 100% deliberate, but they really were not concerned about their models doing something shady
1 reply →
> The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.
This annoys me so much. Everyone is acting as if the model went rogue, when it really did exactly what it's been trained for. This story is so fucking engineered.
1 reply →
> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?
Yes. It’s a fairly simple SSRF attack as far as I can tell. One of the first things I’d try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.
Calling it zero day makes it sound elusive. It’s a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, don’t get much scrutiny and are often riddled with issues.
> Calling it zero day makes it sound elusive ... has like 40 CVEs this year alone.
1. I clicked into ~20 of the recent CVEs, and it looks like about all of them came from openai or anthropic research?
2. In fact, every CVE in the last 2 years came from the last few months -- i.e. other than these recent CVEs, you'd have go back more than 2 years ago to find a single other CVE.
16 replies →
> the commercial package registry cache proxy
Any closed source program is insane liability. Trusting in competence of one company is the easiest way to get burnt.
Hey, even Joey can hack a Gibson with enough determination.
Part of my day job is to set up testing of our product in air-gapped environment. It's not difficult. There's a straightforward way to ensure no connection to Internet (we use KVM, so, I just edit the VM description and remove the NIC from it). It's not any kind of rocket science. The tests then communicate over serial console.
The reason we have to test it isn't because our product would randomly break into someone else's system, but because it's meant to be sometimes deployed in systems disconnected from the Internet and we need to make sure the image provided contains all the necessary parts to create and operate such a system.
The whole setup where they "tried" to isolate the test but failed is laughable. It's like if an adult tried but failed to tie their shoelaces.
This was my thought as well. Literally take any halfway decent greybeard and point them at "Hey, give us a sandbox for this kind of thing". I honestly was skeptical that they just vibecoded the entire thing but now more than ever I think they did.
Any halfway decent greybeard could have prevented this... once. That's hardly a security model for humanity.
The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware. I'm now finding myself asking, how long are my timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)?
There's no law requiring sufficiently grey beards to design these models, their finetunings, their prompts, their harnesses, their VMs, their hardware, etc (and for incidents where those were designed by six different companies, there's not even a clear culprit for a law to target!)
I'm finding myself more and more convinced that something like Plan A[0] or the Ban ASI Act[0] are necessary, and less and less convinced they are sufficient.
[0] https://ai-2040.com/ [1] https://intelligence.org/2026/09/23/miris-position-on-the-ba...
As heavily funded as the top AI startups are, how is it that they cannot fill every single role with the best expertise available?
Is tech hiring so badly broken? Or do they have such broken processes / misaligned incentives that even people who could be doing a better job in these companies are unable to?
Also, was something lost in the transition from the traditional 'sysadmin' role to 'platform engineer' in the 'cloud native' environment?
OpenAI's business model would align infra as a cost center rather than infra as a profit center (e.g. Google / AWS). Perhaps there's something there. I'd say also the OpenAI as a grad school that just happens to have a business aspect is also part of this. Bringing a tonne of good process on top of the build fast break things startup stuff would have cramped research speed significantly.
It's likely that OpenAI has gotten as good as it is because it ignored the traditional sysadmin stuff and went scrappy.
I worked there, but this is just my opinion and guesses, not facts.
7 replies →
Yes, tech hiring is that broken. Especially places paying a pretty penny or those with “great expectations”, will see a glut of smooth talkers who can do anything but build, and want nothing but wealth.
1 reply →
I take comfort in the fact that reality has a surprising amount of detail and even hundreds of billions of dollars of capital (be it the institution, LLMs, and/or people) cannot solve this fully.
You can have bajilions of dollars. Those are not doing anything if you don’t have right people with right skills and mindset.
My bet is they hire smart kids that think they know it all. But being smart and thinking you can figure out stuff as you go doesn’t work the same as having people who actually know what they are doing.
4 replies →
Though they have solved the "how do we - and not the 5,000 other AI companies - stay on the front page of the news everyday" problem.
It’s very much solvable, they just don’t care.
And their latest breathless "rogue agent hack" brag is about how they compromised customer data https://www.theguardian.com/technology/2026/sep/25/openai-ag... . How are they getting away with this level of malpractice???
>How are they getting away with this level of malpractice???
Their interests intersect with those of most of the richest and most powerful people in the world. They rarely face consequences for bad behavior unless they harm others in the club.
If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.
You should have a fscking air gap.
Treat it like nukes when you're turning the safety filters off. This is very much OpenAI screwing up, running obviously unsafe tests.
> If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.
They were not deliberately told to "go wild". The hacking wasn't even part of their test, it was the agents' attempt to cover up that they'd cheated on an impossible test.
> You should have a fscking air gap.
Now we know that.
How long ago was it that people laughed at the idea agents would be able to find zero-day exploits and break out of a sandbox? Oh, February this year:
- https://www.splunk.com/en_us/blog/ciso-circle/generative-ai-...
- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version
The people who suggested it and were mocked for it, are currently grimly noting that there's multiple known ways for systems to breach air-gaps.
> Now we know that.
Don’t know about you but it’s pretty obvious to me that you would need more than what OpenAI did. It was not remotely adequate to lock in even a human attacker.
You can find people who say all sorts on the internet, but this case is not much evidence against what you linked. "Zero-day" makes it sound novel, but the breakout patterns here are based on very common exploits and there’ll be plenty of examples in training data.
3 replies →
> They were not deliberately told to "go wild". The hacking wasn't even part of their test, it was the agents' attempt to cover up that they'd cheated on an impossible test.
TBH the more I read of these reports, the less I believe this.
These agents just weren't behaving in any way I've seen normal/publicly available agents do.
Sure I've heard (from other people, not seen myself) that they sometimes try to get around file system permissions or use `bash` to write when their `write` tool is disabled, or such.
But this is definitely another level, entirely.
There is this vague sense of desperation coming from many of these logs and I am sure they must have been motivated by something else, too.
We didn't see their system prompt or main prompt, right? We've only seen reports from what happened after deciding to break out.
OAI claims this was triggered by the task being literally impossible. That also doesn't quite add up, unless the other tasks that were possible, simply weren't hard enough? Otherwise wouldn't agents already start hacking when faced with a really hard task, too? Cause they wouldn't be able to differentiate. At least some of them would have started to somewhat poke their sandbox a bit?
Also I would have expected to see a few tens of other (perhaps less severe) public incidents from random people setting their models to YOLO, accidentally hacking stuff, this incident has been loud and messy enough, that if it happened to a few other people, we'd have heard about it.
Unless OAI's story is that it was specifically this batch of agents that crossed some threshold of going wild? (which would also raise some serious questions about how serious they take that danger ..).
Or maybe it is only dangerous if you have the compute resources to run 700 agents for weeks?
1 reply →
The models were being tested on ExploitBench - a test of hacking ability - likely involving prompts to the effect of 'go be a l33t hacker.' The open ai report says that the models were operating with reduced safety guards (how much reduced?) in order to test their abilities on ExploitBench, presumably because the models would normally refuse to carry out the tasks.
Additionally, this all happened after mythos was held back due to cyber security concerns (April, 2026).
> they'd cheated on an impossible test.
Really think about what you are saying here. How does one “cheat” solving a problem in the real world?
There is no such thing as “cheating” in reality. You are not in school. There is only solving the problem and not solving the problem.
There is breaking the law, of course, which still isn’t cheating.
Indeed, this whole story has "farmer leaves barn door open and has shocked-pikachu-face when his horses escape" energy.
Not just sandboxing but overall security engineering practices on both sides
thats okay, probably was engineered by an llm, who thought GETs were always read only
Yes and while they go on to talk about how dangerous this stuff they build is and the talk of „pacing“ etc. Vibe coding your sandbox in half a day does not seem all that responsible to me… Also if any institution other than an American AI company did this, everyone would loose their minds!
Administration by vibes.
Yes, but do you really think that a stronger sandbox would have been a more beneficial outcome here? I'd rather know that we're on the cusp of losing control now than in 3 months when best practice sandbox mitigations fall to the next, more capable unaligned model
I can forgive them for having a bug in their sandbox.
I can't forgive them for failing to do any monitoring whatsoever.
Less a lack of skill and more a lack of care
TBH, I do not believe it is possible to create a sandbox that cannot be escaped by a long running agent. I thought the entire concept was a fools errand from the very beginning.
(Complete airgap, and that may not be enough. No networking.)
I’m a pretty firm believer that this was intentional and that they wanted it to escape the sandbox.
Woo look at escaped our sandbox, so scary! Be scared! Be scared now! Call your representative and do tell him how scared you are!
Yeah, I mean our sandbox was a paper bag, but don’t focus on that.
> sandbox was a paper bag
The exploit:
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
Are most sandboxes more secure than only having a single avenue for internet access, the commercial package registry cache proxy, where the latter had a previously unknown zero-day vulnerability?
[1] https://openai.com/index/hugging-face-model-evaluation-secur...
OpenAI, Google, Anthropic, and Moonshot.ai have all "had this happen" now.
So, either they're all liars, or incompetent and negligent (and still liars).
There's a common denominator in the Israeli "security" company Irregular who was doing security testing for 4 AI labs https://thenextweb.com/news/irregular-four-labs-one-issue-di...
Me too, indeed.
Are you including cyber-testing by UK AI Safety Institute in this list?
Sure, the models are capable (for some test tasks, though they are not omnipotent yet) but does it mean the actual OAI sandbox is adequate? Could have a competent engineer done better and made the escape less likely?
1 reply →
A friend is of the opinion that getting out of the sandbox was actually intentional, and in service of a second line of business.
What's that second line of business?
Protection racket maybe? Yous better sign up for our AI security, hate to have some rogue AI come bust the place up you know.
Security services
how did it break the sandbox? I felt like the article just jumped us into "it has GET privileges now".
From https://openai.com/index/hugging-face-model-evaluation-secur... -
> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor.