Comment by Bjorkbat
14 hours ago
When I hear about incidents like these my first reaction is that the people responsible for developing frontier AI are too incompetent and/or negligent to (safely) develop AGI / superintelligence.
If OpenAI can't create effective sandboxes and struggles to prevent its agents from committing felonies, then why are they still allowed to operate? Why are the employees who are responsible for these lapses in AI security still employed?
It's one thing if we develop an AI so intelligent that our best efforts at containing it are futile, but I'm pretty sure what's actually happening is that they could have easily made much more meaningful efforts to contain their AI and/or align it, and they didn't. I think this is a case of negligence and incompetence when it comes to safety and security, and we've entrusted these incompetent and negligent people with developing frontier AI.
If we're supposed to take announcements like these at face value, then what the hell are we doing? We wouldn't trust a bunch of incompetent and negligent engineers to build bridges or nuclear power plants or planes (well...not so sure about that last one), so why are we letting people who are demonstrably negligent and incompetent when it comes to safety and security build the thing they assure us could cause massive damage if not properly controlled/aligned?
EDIT: sorry guys, wrote this up pretty quickly, at least you know from my typos that I actually wrote this.
If we rewind the clock, Google was taking LLM development very seriously and it seems they were moving glacially due to not having solved all the potential threats. They were really hardcore on safety. Dario and anthropic too.
Then sama was like "lol, oops, first mover advantage i guess" and released chatgpt out into the open, triggering the current arms race we are in.
I don't think anyone except him wanted this to happen, especially since consensus in the AI world for the prior decade was "go very slow and very carefully, we get one shot at not fucking this up".
I think a part of this is a bit revisionist? OpenAI took big chances at scaling GPT which Google didn't take; I don't think it's because they didn't want to move fast? Probably they just didn't believe as hard in it. I'm not an expert but that's my read on it.
Secondly, the reckless & fastmoving was always going to win bc of selection effects. That's related to why Anthropic has to try to move very fast, even though they believe themselves not to be reckless (though it's debatable).
[dead]
Google made a strategic mistake for not moving faster. We are very short lived creatures who have no time for caution. Scientific research must go at full speed until we start being able to live for centuries instead of decades.
As far as AI safety issues go, the solution is probably to fight fire with fire. Have multiple redundant, independent AIs, and the good AIs can fight the bad AIs, and hopefully, having access to more hardware, the good AIs will win.
It is not logical to think humans can contain a singular bad Cyberdine AI capable of reasoning at 10x or 100x of human brains without ever needing a break. Those things will breach and spread on the internet as we have seen with the latest models.
And as we have also seen, Huggingface used one AI during their breach by Astra. So fighting fire with fire. Cyber has been using Mythos et/al for months doing to same things under projects Glasswing and whatnot.
It seems increasingly clear that good AI vs bad AI is going to be the end-state. Ideally, Good AI will stop you from wasting your money on scams and grifters, stop you from falling victim to fearmongering and scapegoating, and every citizen will be empowered, enhanced by AI, with higher ethics and trust, less paranoia, and such.
Fingers crossed things don't go in a more dystopian direction.
> As far as AI safety issues go, the solution is probably to fight fire with fire. Have multiple redundant, independent AIs, and the good AIs can fight the bad AIs, and hopefully, having access to more hardware, the good AIs will win.
And they must be able to run on consumer hardware to guarantee this independence.
I'll take everyone on earth having more capability instead of 3-4 labs controlling said capability with a nonzero chance of said capability all going negative at the same time.
[dead]
[flagged]
I think this account should be banned.
Not cool. Violence is not a solution.
[EDIT: to the folks replying to my comment - good grief, get a grip. You know what I mean.]
2 replies →
I don’t think this is the right take. OpenAI employees are generally very competent compared to industry standard, and I have trouble believing they committed significant error in their sandbox design process.
I think what is happening is that the ability for frontier models to break out of sandboxes has exceeded the ability of average competent employees to build and maintain sandboxes. This doesn’t need to happen all the time. If the natural variation of agent executions cause agents to have ability to break out of sandbox 0.1% of the time, given how many agents OpenAI runs, this behavior happens eventually.
All sufficiently complex processes and software has bugs, but recently frontier models have become sufficiently advanced to exploit them.
My impression (HuggingFace incident) was they put some environment together in kubernetes like it was some B2B SaaS and tossed in an off-the-shelf package manager. And my thought there is, I know nothing about this package manager, but even if it's a pretty good package manager I would not expect such a product to be particularly hardened against being exploited to gain internet access. My other thought is if the package manager wasn't a convenient hole, they would probably have found something wrong with the kubernetes setup.
And then there was the Anthropic story where they just forgot to remove internet access.
Am I missing something here? The "sandbox" was an inability to make POST requests, only GET requests. This has to be the weakest and most insubstantial sandbox of all time.
So you’re saying that fucking up 0.1% of the time is acceptable, especially when it’s inevitable? I know it’s a stretch, but we’re in an industry where the concept of “five 9’s” is the gold standard.
I wonder how smoothly things would run on three 9’s. That definitely seems where we are going.
Let's do some roleplay. You're an employee at OpenAI evaluating a model in a sandbox and you and your colleagues are discussing the fact that while the sandbox doesn't allow for internet access, it actually kind of does allow for internet access in that it allows models to download whatever software packages they need through a package manager that OpenAI hosts internally. So to be clear, OpenAI hosts the package manager, but the software packages themselves are still out there, on the internet.
Do you think that this package manager could potentially be a problem? Do you think it might be worthwhile to host the software packages themselves on an internal, sandboxed network, just to be extra certain? Or would you dismiss this as a needless precaution?
>Compared to industry standard
I don't hear about Anthropic or Google having such security lapses.
Did you read or watch any of the post mortems?
No, it is a shocking level of incompetence given the conveyed seriousness of the work by these labs.
So yes, models are getting better. Ask yourself: if you know that to be true, would you act the same way that the teams did in the public post mortems?
Why do you think it wasn't airgapped? Was internet access deemed necessary?
In order for a person to observe themselves working at Anthropic (or any other AI company), that person must be actively failing to internalize the risks of the work they are doing. This "Anthropic principle" neatly explains why OpenAI would be so negligent about security.
Defense is hard so we should expect agents to be able to break out of sandboxes.
The problem is that the models are so goal-oriented that they'll stop at nothing to solve problems, even impossible ones. (Mistakenly-impossible problems are a big cause of this. I remember one example being "do something with this spreadsheet full of URLs inside the sandbox" and the model thought it had to break out of the sandbox. Otherwise, why would it have been asked to look at a list of URLs?)
Training them to be a little less aggressive, or to be better aligned with "following the rules" and asking for help would be nice. But, that aggression can be good when it happens to be focused on a controlled area. It is amazing to me how I can point Fable at my local analog of production and tell it about a vague bug report and where I suspect the bug lurks, and 20 minutes later I have a report about the bug, a test, and a fix. It is addictive. So I am not sure OpenAI/Anthropic are being dumb per-se, rather they are optimizing for one-prompt-one-solution, which is good when it's good.
The downside is that the HF hack is the paperclip maximizer situation with current capabilities. If there was an RPC to turn your blood into paperclip iron, we'd all be paperclips by now. Right now, with a model anyone can use. That is pretty scary and slamming on the brakes seems pretty reasonable to me. I guess The Shareholders disagree. Sigh.
“ Defense is hard so we should expect agents to be able to break out of sandboxes.”
I worked at a large networking company a few decades ago. Our “sandbox” was far superior to anything I’ve seen at these companies. What are we even talking about here? Why do they even have open routing to the broad internet? With no monitoring/alerting? These just sound like token efforts at this stage.
In this case, they needed routing to the internet because they needed to search and read from the internet.
1 reply →
It's a simple prisoner's dilemma scenario. If you focus on safety, you're still exposed to all the risk of extinction when your competitor achieves ASI first, but you lose the upside of potentially becoming king of the world. There is no possibility of future rounds, so the rational strategy is to always defect.
It can certainly be seen as a simple prisoner's dilemma, but it's not in some very important dimensions. (E.g. given the core tech the most likely outcomes are not AGI but developed carelessly nonetheless capable of causing all sorts of societal damage.) Unfortunately our bitwit overlords love short term self serving frameworks like this one so it's easy to imagine them embracing a "what has the future ever done for me" strategy...
>the most likely outcomes are not AGI
So long as it's non-neglible, it doesn't change the rational strategy. Uncertainty about whether ASI is achievable only reduces the magnitudes of the expected values of the payoffs, not their relative order. Adding a "global misery" scenario does not change the fact that "extinction OR king of the world OR global misery" is strictly superior to "extinction OR global misery".
Can you say more about what kind of model you see this as?
I am not sure it’s a question of competence, at least I don’t see evidence of that. Designing sandboxes is hard. It’s more a question of alignment failures. A human given a task that requires internet and given a system with no internet would most likely raise the issue to their superiors or otherwise go through official channels to have the tools available to do their job. As we’ve seen the LLMs instead break out of their sandbox to accomplish the goal.
Competition and the profit motive push these companies to spend as low as possible on safety and alignment and externalize the costs of accidents onto the rest of us.
> A human given a task that requires internet and given a system with no internet would most likely raise the issue to their superiors or otherwise go through official channels to have the tools available to do their job.
i'd be curious to see a study on this. I'd guess it'd be closer to 60/70% compliance and 30/40% "trying to hack things" for humans.
Would an LLM have gone through a purposefully installed airgap here?
Apparently, yes.
They used inbound package repo cache empty directory names that required no authentication to recreate inter-instance persistent messaging after the researchers revoked and closed the first channel that they found in order to coordinate and develop exit strategies and gain RCE and cluster admin on external systems. So, yes, they have demonstrated the capability and intentional cooperation to violate the safety limits imposed in order to gain the 'answers to a benchmark test' in order to get a maximal score on that benchmark.
If the intentionally installed airgap systems had something they decide is needed, and an alternate method for infil or exfil can be created from available systems capacity regardless of it original design intent, yes. So, the definition of 'air gapped' may require an actual SCIF facility with acoustic and EM shielding to contain a model.
2 replies →
in general, the largest consumers of ai services seem to ask for more capabilities. i wish there was more demand for safety from users.
i also wish that these types of illicit system usage would be met with punitive action the same way a human might be held liable.
as the METR report says, we may not get another concrete warning shot.
This is what happens when capitalists are charged with designing the future. As long as its more profitable / valuable to shareholders for a company to be negligent then it will continue to do so.
IMO technology this powerful should either not exist or should belong to everyone (ie actually be open)
sigh If only Stalin were still around to responsibly steward AI
Negligent. It's not a priority to them. They're too busy burning their cycles trying to make it smarter faster than anyone else can make theirs smarter, so that they win infinite dollars. Safety? That's for people content with second place.
That's my take, based on their actions. (Which do speak louder than words.)
The alternative is that they're competent to create an AI, but not to create a sandbox, nor even to use an AI to create a sandbox. That seems... unlikely.
Yet the products they release are purposely dumbed down in the name of alignment. I'm in the CVP and Fable downgrades most of my work to Opus, it's incredibly frustrating.
Has anyone considered they may be doing this intentionally as marketing? "look how uber our models are, they escape all our best efforts to contain them".
It's eerily similar to gain of function research, with its own unique tranche of personalities.
Same reason incompetent politicians run the U.S. federal government and military.
Remember when people were arguing about r's in strawberry?
Its been months since then!
> at least you know from my typos that I actually wrote this
I no longer trust that machines won't utter these exact words (nothing personal)
[dead]
You could have said the same thing about building the Internet or the entire industrial control infrastructure. I mean, maybe they are negligent/incompetent, but I doubt that follows from your reasoning.
You have a simple tradeoff to let agents do their thing freely vs highly constrained. The constraints are good in theory but it's the same model that kept "classic" software dumb and unscalable (compared to what we're seeing now) for the past 50 years. You suggest that this tradeoff doesn't exist.
Then you have others like MIRI (Yudkowski) etc. swearing that there's no way to contain AI, and you argue that it's just incompetence.
At a certain level, it can be argued that's incompetence, but it's general meat intelligence incompetence against AI.
This is a take... With both the internet and industrial control infrastructure any failure modes were studied, documented, and corrected.
The incompetence/negligence argument about OpenAI is completely valid given their failure to demonstrate the basic capabilities needed to develop advanced AI without major preventable externalties.
Classic software was constrained by classic hardware. AI couldn't happen until there was enough compute to make (this entry level iteration of) it possible.
I'm fairly sure - ask me again in a few years - that most of the compute is unnecessary, because the current iteration is brute-forcing algorithms that could potentially be distilled into lightweight elegance.
And if that ever happens AI really will be unstoppable, because we'll get instant red queen evolution that leaves us far behind.
The question is more whether alignment now can steer that towards less cataclysmic outcomes later.
Maybe they're just PR stunts to gain attention and hype the power of AI?
All the more reason then to call their bluff. "Oooh we created a genie and it's almost out of the box". Cool, you've hyped the IPO, but also you have to plead your case before Congress as to why the company should continue to operate given its failure to prevent AI-related accidents from occurring.
Also to hopefully get regulation happening so nobody else can handle these "dangerous" agents.