Comment by root_axis
4 days ago
The fact that they're using Artifactory in the first place is already a failure. The fact that they didn't airgap the network or even use any type of egress blocking is just absurd.
Zero days are nothing new, secure containment has been a solved problem for a long time.
I'm curious how you would airgap a datacenter-scale training run. No reading training data from a cloud store, someone from OpenAI flies out to Abilene or whereever with a suitcase full of SSDs? When researchers in SF write code for a new RL environment, they ping somebody who sticks it on a flash drive and walks over to the server rack with the GPUs? Has anyone ever trained an LLM like this?
Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...
Data diodes exist and are common in high-security environments https://en.wikipedia.org/wiki/Unidirectional_network
The hardware literally only goes one way. Nothing can "escape" without rewiring cables.
Neat, never heard of those. I'm curious how this works with, like, acks, and also whether there's any public prior art for anyone having trained an LLM beyond one.
exactly, how does a trillion dollar valued company not consider this? they seem intent on having a "rogue" incident happen
> how you would airgap a datacenter-scale training run
You could start with standard firewall rules and VLANs. Pretty basic stuff we've had for decades.
That's not an airgap though.
As someone who has actually had to maintain multiple airgapped networks in the past and knows the amount of work that's required for that, especially at the scales of data they're working with...
All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd. Especially in the same breath as claiming the models apparently aren't intelligent enough to warrant such concern.
Nah. It really wouldn't be hard to set up an environment where models can't damage the outside world from. The fact that people still don't think this should be the norm or paint it as some technical impossibility is mindblowing.
People execute, study, develop, and test literal malware of every caliber everyday without issues. Yes the way they do it is requires care and in restricted settings. Yes it costs money and takes time to do that. Yes the people who do it are required and expected to do that.
Oai and anthropic are more or less run by clowns who intend to profiteer off of regulatory capture and who knows maybe directly from the harm their technologies can cause.Because of these takes we have to wait while they try to expand their verticals into the pain and suffering industries. Palantir is already pitching smoking cigarettes so there's that.
Well, as someone who has had to maintain multiple such airgapped networks you are obviously in the know that this could be done, because you've done it before.
The point here is: they didn't even try.
They could have easily temporarily cut off the segment that this was running on, or restricted and monitored its egress to the point that a hacking attempt on an outside party would have been detected.
The point here is (again): they didn't even try.
So yes, they obviously should have either done that or they should have not run their experiment given that they are claiming that the models are intelligent enough warrant such concern. In fact they post one article after another about how much smaller and less capable models are intelligent enough to warrant such concern and in fact warrant regulating this domain to the point that only they are able to own and operate this stuff.
Let's make a biology analogy: if you can't safely operate with harmful substances that doesn't give you a license to do so in an unsafe way. Then you just don't do it.
> the amount of work that's required for that
I find it hilarious that you'd balk at doing work to secure a network. Even funnier for a company that's supposedly solved white collar labor.
You misunderstand me. The work isn't primarily in "securing" the network. It's the hassle involved in the continuous transfer of data from one side to the other (and for a proper air-gap, that transfer only goes one way, which is laughably infeasible here). In my line of work that's mostly software security patches and weekly code transfers, but obviously these labs have multiple orders of magnitude more.
And I don't know how you expect an LLM to do airgapped data transfers. The fact that you even bring that up makes me think you don't actually understand what airgapping entails.
3 replies →
Things that are not intelligent could still warrant very strong safety measures.
Depending on the risks associated, the complexity and effort of certain containment measures would also not neccessarily be something that could be used to brush them aside.
It's always going to be too hard as long as companies aren't held accountable.
1 reply →
These companies are taking millions of dollars, sucking in experts everywhere they can, and that is before they even hit the stock market.
I don't think it is too much to ask that they devote their resources to creating an air gaped campus for testing. If they need to build an entire data center campus with copies of the pilfered data they used to train the damn things, then so be it. HF and other incidents have proven that trying to block models off by software networking is a dangerous game when they can just exploit some unknown flaw in a library nobody knew about for over 40 years.
The inconvenience of implementing correct safety measures is part of the cost of undertaking certain activities. When an activity's ratio of cost versus benefit is too high, there's no way to juice the benefit, and the last place to cost-cut is in safety measures, the responsible thing is to refrain from the activity in the first place.
> All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd.
I'm sorry was this meant to be a defense of Anthropic? The company claiming A.I. is an threat to the human race and only they can be trusted with it?
How dare we demand more from these trillion dollar companies. Good thing they've got PR drones around the web excusing their incompetence.
They aren't operating at scale then
But an air gap is a gap of air. That gap doesn’t exist if the rack is plugged into Ethernet.
Like a castle with a moat. If the thing gets attacked you can’t say “we had a moat!” if the drawbridge is down.
[flagged]