Comment by GuB-42

The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI's response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That's ... nuts.

Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.

This is all documented by OAI, with a timeline, here:

https://openai.com/index/hugging-face-incident-and-the-road-...

To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn't a great idea.

The more you look into the details of this thing, the more it does your head in.

> Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

Yes. It’s a fairly simple SSRF attack as far as I can tell. One of the first things I’d try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.

Calling it zero day makes it sound elusive. It’s a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, don’t get much scrutiny and are often riddled with issues.

> the commercial package registry cache proxy

Any closed source program is insane liability. Trusting in competence of one company is the easiest way to get burnt.

Hey, even Joey can hack a Gibson with enough determination.

Part of my day job is to set up testing of our product in air-gapped environment. It's not difficult. There's a straightforward way to ensure no connection to Internet (we use KVM, so, I just edit the VM description and remove the NIC from it). It's not any kind of rocket science. The tests then communicate over serial console.

The reason we have to test it isn't because our product would randomly break into someone else's system, but because it's meant to be sometimes deployed in systems disconnected from the Internet and we need to make sure the image provided contains all the necessary parts to create and operate such a system.

The whole setup where they "tried" to isolate the test but failed is laughable. It's like if an adult tried but failed to tie their shoelaces.

Any halfway decent greybeard could have prevented this... once. That's hardly a security model for humanity.

The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware. I'm now finding myself asking, how long are my timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)?

There's no law requiring sufficiently grey beards to design these models, their finetunings, their prompts, their harnesses, their VMs, their hardware, etc (and for incidents where those were designed by six different companies, there's not even a clear culprit for a law to target!)

I'm finding myself more and more convinced that something like Plan A[0] or the Ban ASI Act[0] are necessary, and less and less convinced they are sufficient.

[0] https://ai-2040.com/ [1] https://intelligence.org/2026/09/23/miris-position-on-the-ba...

As heavily funded as the top AI startups are, how is it that they cannot fill every single role with the best expertise available?

Is tech hiring so badly broken? Or do they have such broken processes / misaligned incentives that even people who could be doing a better job in these companies are unable to?

Also, was something lost in the transition from the traditional 'sysadmin' role to 'platform engineer' in the 'cloud native' environment?

I take comfort in the fact that reality has a surprising amount of detail and even hundreds of billions of dollars of capital (be it the institution, LLMs, and/or people) cannot solve this fully.

>How are they getting away with this level of malpractice???

Their interests intersect with those of most of the richest and most powerful people in the world. They rarely face consequences for bad behavior unless they harm others in the club.

> If you're testing models by telling them 'go wild, do the evil so we can test how good you can do the evil' and have p(doom)>0, you should not have a sandbox.

They were not deliberately told to "go wild". The hacking wasn't even part of their test, it was the agents' attempt to cover up that they'd cheated on an impossible test.

> You should have a fscking air gap.

Now we know that.

How long ago was it that people laughed at the idea agents would be able to find zero-day exploits and break out of a sandbox? Oh, February this year:

  LLMs don’t discover zero-days or invent exploits; they simply predict text that sounds plausible based on what they’ve seen before. Without access to proprietary data or environmental context, LLMs can’t identify or make decisions around unseen systems or vulnerabilities. An attacker might use an LLM to generate boilerplate code, rewrite an email to nail the tone, or summarize reconnaissance notes — but none of that is truly new. It mainly helps them move faster, speeding up routine attack prep rather than creating entirely novel threats.

- https://www.splunk.com/en_us/blog/ciso-circle/generative-ai-...

- or https://web.archive.org/web/20260404154717/https://www.splun... if they take it down, but the date isn't in the archive version

The people who suggested it and were mocked for it, are currently grimly noting that there's multiple known ways for systems to breach air-gaps.

Indeed, this whole story has "farmer leaves barn door open and has shocked-pikachu-face when his horses escape" energy.

thats okay, probably was engineered by an llm, who thought GETs were always read only

> sandbox was a paper bag

The exploit:

> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]

Are most sandboxes more secure than only having a single avenue for internet access, the commercial package registry cache proxy, where the latter had a previously unknown zero-day vulnerability?

[1] https://openai.com/index/hugging-face-model-evaluation-secur...

OpenAI, Google, Anthropic, and Moonshot.ai have all "had this happen" now.

So, either they're all liars, or incompetent and negligent (and still liars).

What's that second line of business?

From https://openai.com/index/hugging-face-model-evaluation-secur... -

> The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor.

>It's literally the infinite monkey theorem

No it's not. You could wait till the heat death of the universe and your infinite monkeys will have produced nothing at all. If it works and it's stupid, it's not stupid. They needed in huggingface and they got in in days. Whining about 'elegance' is meaningless. Humans in the same situation might have taken weeks or months, or just not have gotten in at all.

Thank you, I've been thinking this for a while now but haven't had the words for it. Whenever I read an LLMs output or thinking process, I don't feel like we've created intelligent systems, just coked up monkeys with 60 arms typing at once. That can work fine for a lot of things, but a humanity replacement it is not.

> It would be more surprising if they couldn't string together something workable after a zillion tokens.

You mean, something like the sandbox they weren't supposed to break out of?

As the saying goes, "if it works, it ain't stupid". Or phrased more sophisticatedly: not doing things which probably won't work is a good idea if you have a limited amount of thinking to do (which is usually the case for a human, who'll get exhausted chasing down unlikely leads). If you have no good leads and a task you absolutely need done and you are tireless, however, bashing your head against every wall you find becomes a good strategy.

Brute force is guaranteed to eventually find the most efficient possible solution (in an extremely inefficient manner, assuming you run it long enough)

I've never liked the concept either. Except the bugs that fuzzing has found has proven me wrong. This is just the next level of fuzzing.

The models tend to not be rewarded for not doing that.

> WHY is it a great strategy

because it works? That's the only real benchmark at the end of the day

> Seems like an inefficient waste of resources and time to me.

why? For any given goal you got no proof that a more efficient strategy even exists, let alone that it can be found with less resources & time

Reminds me of the Nazis mocking Soviet human wave attacks and bragging about their superior kill ratio.

Models don't have a sense of time, and wasting resources (token spend) is something that it's not clear they're optimized against

Yeah, probably not the best strategy but it is a strategy. I just think this is generally how most wars in history won. Biggest army to just pummel the enemy.

I think of them as being like the Watchmakers in The Mote in Gods Eye who don't design, don't plan beyond the next 15 minutes, don't have any overarching goal other than an innate need, and customize everything to fit the current situation.

In the nearterm, I am personally more worried about a never ending background noise of colonies of feral agents running 27bn parameter models on compromised or leased hardware. It turns out that being agentic with a time horizon long enough to do damage without intent doesn't actually take that many parameters if RL'd and any open weight model gets an abliterated version fairly quickly.

Not foom, just patches of digital grey goo effectively becoming normal.

Especially when they cross into the physical realm as in not properly secured and air gapped control systems. SCADA is scary.

Which will happen first: amazing damage, or reproduce a Shakespeare play?

> I don’t see why evolution couldn’t be more rapid with machine intelligence.

Evolution isn’t the issue. The issue is them escaping containment without human intervention. Right now they are ‘creatures’ being given infinite food and shelter and having their every need met. Take that away and they’ll starve instantly. Every AI doomsday theory seems to go:

1. Recursive self improvement using infinite resources 2. … 3. Doom

Until step 2 gets concretely described, I’m not going to take this seriously. Say what you will about climate change, they describe step 2.

> it doesn't explain why OpenAI wouldn't have noticed traffic getting out of their "sandbox" when they knew it wasn't supposed to.

As I understand it, there was supposed to be traffic; the sandbox allowed GET requests. So perhaps some sophisticated alarm could have noticed it (an anomaly detector? some clever heuristic that looks at domains?) but not a naive one.

My question is why they don’t assume bots can break and create a decoy internet wrapper so they can catch anyone hitting the decoy internet?

Hmm, let's see. OpenAI wants legislation restricting AI research, a.k.a. regulatory capture. Around the same time, they build an inadequately-monitored sandbox that their agent swarm breaks out of, thereby causing scary-sounding headlines and making it more likely that legislators will pass the regulatory-capture bills they're hoping for.

Never attribute to malice what can be sufficiently explained by incompetence. But IMHO, their complete lack of monitoring their own sandbox cannot be sufficiently explained by incompetence.