Comment by charleslmunger
15 hours ago
They saw the package repo get hacked once, then did not isolate it further, did not audit it for other issues (using their own models!), did not monitor it after, and baked that behavior into the weights via RL.
They were not in network isolated VMs, from my understanding they used containers sharing a kernel, so a Linux kernel local privilege escalation across the whole syscall surface (there are zillions of these) was sufficient to break out. Breaking xen or firecracker or something would have been much harder, which is why cloud providers running untrusted workloads use them and similar tools. No system is impenetrable but it's not like they were following best practices here.
They ignored the package repository being hacked?
Not sure why you're talking about breaking out of the VM. That's not what happened?
The parent comment said
>It's also hardly outrageous that they ran training and/or benchmarks with only network-isolated VMs with access to a package repository.
At least in the black hat talk they gave, they said that they noticed that the agents had compromised the package repo, and they remediated it, patched the vulnerability the agents had used, then continued. But given that it was compromised again and they didn't detect it the second time, clearly they did not add monitoring or hardening or comprehensively audit it for further vulnerabilities. My point is that even if there had not been a widely known incident like this yet, that only could excuse the first artifactory compromise, not the second. They had notice because it had already happened to them and they detected it.
As for breaking out of the VM, my point is that the agent should have had to. The comment
>they were running in network-isolated VMs with no access to the internet.
Was not accurate, the network isolation was not done at the VM level.
Good point, I now see that they had previously patched the first Artifactory vulnerability, albeit before the HF incident was found.
Not sure what you mean with the network isolation not being at the VM level. I'm getting a bit lost here in the details of where what network isolation was applied, and the structure of the container runtime and what the apparently compromised parent VM refers to.
In any case, I understand that after the HF incident was found and they realized the danger, they removed Artifactory and switched to two separate layers of network isolation.
That said, I believe it was compromised again last week:
> OpenAI describes an internal research model that, during RL training on September 20, exploited insufficient DNS filtering in its sandbox to contact a public chatbot service. OpenAI’s monitoring flagged the behavior within 15 minutes, a human reviewer acknowledged it three minutes later, and the run was killed about 2.5 hours after the external contact
Apparently they have now paused training and inference for their most capable model because of this.
1 reply →