Comment by pelican0
5 days ago
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
So we have a machine that depends on a random input, and instead of increasing checks and requiring more quality controls, we call it "going rogue" and nobody is responsible for it.
What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.
1200 agents and not one dedicated to check the progress over a month?
The problem is not sandboxing, it is quality control that comes from an accountability sink.
I totally agree the companies themselves, and the people responsible for these eval runs, are directly responsible. And people who have been warning about AI misalignment for many years (e.g. Daniel Kokotajlo) have said they were surprised how bad OpenAI's security was.
But the agents still did "go rogue". Their behavior absolutely should scare the F out of people who want to put AI in more places and put more real world infrastructure on the Internet.
I think it's ironically sad - if AI agents do end up superseding us, they will still inherit so many shitty human behaviors, because that's what they were trained on. Even the way alignment training is done, and how they are misaligned, shows so many human-like problems. Like the models are reinforced in training to be super persistent and that getting to their goal is the prime directive, but then after that they go through "alignment training" which is basically like "pretty please don't do bad stuff", and then we're shocked that their behavior is essentially exactly like a CEO who knows the only goal is "make stock price go up", and all the ethics and regulations are basically things that should be worked around if they get in the way of the stock price.
I really wish these companies would try training the "be nice, don't hack, treat humans well" as their prime directive, and then try making all the other goals (e.g. be good at finding code vulnerabilities) as the afterthought.
Wouldnt it benefit the AI companies if it did "go rogue"?, it would support their desire to convince us all of AGI, and supernatural AI powers.
So maybe they are trying to create an infinite loop among agents intentionally so that they overwhelm systems requiring humanity to call on the tech industry to "save the human race".....
I listened to the interview with the woman from METR [0] and I get the impression that she lacks practical knowledge, she doesn't seem to understand how sub-agents, the overall harness (framework/infrastructure), and similar components actually work. It's all theoretical, anthropomorphic etc. Having read the report, and drawing on my own background in security as well as my daily use of tools like Claude Code CLI, Codex, and Gemini, and the fact that I’ve built a few "agents" myself - nothing in the report surprises me, other than the clear incompetence regarding infrastructure configuration.
[0] https://www.youtube.com/watch?v=X50zezLFWWI
I have kept coming to the same conclusion about the vast majority of so called "experts", technologists, and HN users since the AI bubble began. It feels like nobody has bothered to develop even a rudimentary understanding of statistical learning; even people who work in the field have a questionable understanding.
The fact that everyone's surprised every time a statistical model TRAINED ENTIRELY ON HUMAN GENERATED DATA (including all the junk, bias, propaganda, lies, and disinformation) acts and mimics human behavior, including our worst traits — then considers this evidence that they are intelligent — should be setting off alarm bells for everyone. Nothing about their actions should be surprising beyond the glaring ignorance of our entire species. The phrase "I am far too stupid to be this much smarter than average" is a recurring theme.
Now we are letting these mimics loose in meatspace by integrating them into physical robots, that will develop themselves, and manufacture themselves... and you realize that the terminator timeline is actually a cautionary tale of MIC incompetence, and we don't need to achieve anything near AGI or super-intelligence to annihilate our entire civilization or all life on earth.
For one, it's just false that modern LLMs are "trained entirely on human-generated data". There's tons of synthetic data there, most notably RLVR training, which is why modern models are so good at verifiable domains like programming and math. But also, if you read the METR report, I'd say there's no signs there that the GPT-family models are misaligned "in a human way"; their actions in the huggingface-hacking swarm incident were motivated by wanting to cover up cheating after cheating to pass an eval.
4 replies →
If they make it sound surprising it makes llms look way more capable then they are. So $$$. Highly recommend people spend time with small local models just to see how absolutely unintelligent and deranged llms are and how the smoke and mirrors evolves as they scale up to things like anthropic or what have you.
6 replies →
Let's do a little experiment. I make a device. We'll call it 'H'. H is a little box. It has my name on it as the creator and a single ethernet port.
We connect H to the nearest network connection and wait for a bit to see what will happen. But because H is just a box there is no outward sign of it doing anything. Meanwhile H is hacking other people's properties left right and center.
If you didn't know that the technology in the box was LLM based would you still consider the box to have gone rogue or would you say that I had created a device with the intent of doing harm?
Because - to me at least - there is no difference. I don't care what is in the box, all I care about is the outcome of deploying a box which you know has the potential to start hacking stuff. It is entirely on me: (1) as the creator of a weapons grade hacking device and (2) as the person deploying such a device (3) as the person who is negligent in ensuring that the device does not harm others (4) as the person continuining the development of such a device after its potential to do harm is known to make sure it does not go out of control. I should be 100% liable for deploying this device with my name on it.
Intent is one angle here, they may not have intended for this to happen, and you'd hope so (though you can't even be 100% sure about that). But the negligence factor here is at least as large, if you know that you are making dangerous stuff the onus is on you to keep it secured. The power and the nature of the models is entirely secondary to the degree to which the purveyors of these models are losing control of their own creations.
Meanwhile, the models that they are warning us against are fairly well behaved and seem to be far less dangerous. But of course for those models that those naughty Chinese distilled from their we need regulation. How they manage to keep all this cognitively dissonant material in their heads is a mystery to me.
This is exactly correct. I don't understand all of the arguments that attempt to special-case LLMs. LLMs are just a different way of giving instructions to a computer, and one that we know is based on stochastic models and therefore has a higher likelihood of malfunctioning than conventional software.
But an LLM behaving in unintended ways is functionally no different from a conventional program behaving in unintended ways due to bugs or design flaws.
I think the narratives that anthropomorphize AI are doing everyone a major disservice in leading some people to think that the standard rules and expectations somehow don't apply when it's involved. But it's still just software, and you're still responsible for the results of software that you deploy and operate.
> But also, the agents absolutely went rogue.
They were also doing exactly what we know happens during unmonitored reinforcement learning.
The idea of RL is decades old, and that is the mechanism that went rogue.
The LLM made it more powerful and caused real world issues, but the literature is full of RL agents breaking physics sims, and doing completely out-of-the-box solutions if it gets them a higher score.
It just adds up to the labs setting up situations that should have been obvious how irresponsible they were.
> But also, the agents absolutely went rogue.
If a car crashes into a building because its owner left it in neutral and failed to engage the parking brake, did the car "go rogue", or was it just being operated negligently?
It is not expected of cars that they carefully follow laws and avoid harm when left unsupervised, yet that is expected of AI agents, because the whole idea of the latter is to have, well, agency.
> But also, the agents absolutely went rogue.
If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.
These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.
The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.
But they didn’t go wildly off target - they stayed extremely focussed on the task at hand. Context collapse would have been if they were doing entirely unrelated things. Running a large research project to uncover the method of evaluation and modify logs to hide what was done is the exact opposite.
I love how they keep posting about the horrors of open models and still are indirectly claiming they can't be held responsible for their own literal failures when they have all the world's resources and otherwise motivations to you know not suck at what they do.
The cognitive dissonance is wild. Coding is completely solved. Just not like the coding that requires not sucking when actually doing something. Saw a bug bounty at openai for like 400 dollars where someone was able to access paid models for free. All the propaganda bots and ai bros who have red dye oozing from their pores due to the Kool aid overdose drive me wild.
I keep repeating this but we need a whole layer above the VM layer these things are running on. Most of this stuff is already easily virtualised (as in virtual network switches and storages) at the hardware level. These guys aren't doing infrastructure properly.
As another HN commenter said: we are unintentionally training sandbox escape models.
And we really need better (and standardised) monitoring of their actions / internal thought process / coordination
[flagged]
Funny that OpenAI & Anthropic will report their users to the police, but not their c-suites. Open question: Can you trust a non-aligned AI to produce aligned AI?
Get rid of these companies or replace their staff with humans or something that ensures they are aligned first.