Comment by simonw
6 days ago
You know this makes OpenAI look really bad, right?
Hugging Face had to tell all of their users, many of them paying customers:
> As a precaution, we recommend rotating any access tokens and reviewing recent activity on your account. If you believe you are affected, or want to report a security concern, contact us at security@huggingface.co.
HF also said this, I'd be very interested to hear how that got resolved!
> Finally, we have also reported this incident to law enforcement agencies.
I could fully see them thinking the incident disclosed yesterday would have made them look good ("wow, OpenAI's models are so capable!"). That it didn't occur to them to discuss specific preventative measures to be taken in the future (airgapping as a foolproof one already familiar to the CTF world, anyone?) indicates to me they're not taking their job seriously; they are the ones treating this as a marketing charade.
It's very difficult for me to reconcile belief in the existential risk business with what they actually did. So I agree with you that this makes OpenAI look badly incompetent; but their communication on this makes me think they don't realize it.
For what it's worth I don't agree with the xrisk-ness of these models; they're dangerous, but almost certainly only temporarily while a new equilibrium is reached via more secure software. Open models are probably an essential part of the recipe (as you noted) for doing so. I also have a personal suspicion that LM-accelerated formal verification will have no small role to play here, sidestepping the cat-and-mouse game of bug finding-and-fixing.
Here's the language that makes me think they are taking this seriously:
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete.
That's not well massaged PR language - that's the kind of thing you dash out when you see a major shitstorm brewing (HF had already publicized the attack before they knew it was from OpenAI) and you want to get ahead of things while you're still pulling together the full story.
I expect we'll find out within a few days if OpenAI are going to keep their promise to "share more details on the vulnerabilities, incident, and findings". If they don't do that I'll reassess how I interpret their initial post.
> We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.
With who? Who are these "defenders"? None of the US labs have done much for the greater good as of... Ever. Of course a frontier provider can leverage their own resources at scale and pull something like this off. If anything this should showcase how dangerous OpenAI and Anthropic are in their current states and maybe the powers shouldn't be concentrated as they continue to move.
I will bet that the RCA debriefed by OAI is going to be a lot of lipstick and very little meat.
> airgapping as a foolproof one
How would the model get any packages that it thinks it needs to complete the task at hand? Not a well-specified task that those tasking it could anticipate and provide all resources up front, but one of discovery.
>You know this makes OpenAI look really bad, right?
The target audience is regulators. They want to look like the smart guys really concerned about AI safety, when they come asking for open weights models to be banned and for other regulations to cement in their moat.
They want this to look like a demon core incident. Bomb and Nuclear reactors still got built.
The lesson OpenAI and Anthropic should have learned from the whole Fable export controls thing should have been "don't pull stunts with the US government".
Turns out they can backfire.
Apparently, the lesson they learned is that their marketing+political stunt worked exactly as planned and they should keep doing what they're doing.
They are still openly lobbying for more AI regulation.
https://www.anthropic.com/news/donation-public-first-action
> You know this makes OpenAI look really bad, right?
Please do explain how this event that makes their product look powerful and perfectly aligns with their openly stated long term goals of pushing for more AI regulation makes them look bad.
It makes them look incompetent, and like they are not up to the task of keeping their AI models "safe".
This very thread is full of comments from people who are shocked at how badly they messed this up.
Whose opinion do you think they care about most? Some random people on HN saying "wow this is a bad look", or the investors who will read dozens of headlines to the tune of "New OpenAI product did something EPIC and INSANE!" and immediately start lining up to throw them a few more billions during the next funding round?