Comment by ammon
5 days ago
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
4 replies →
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
The impossible problems part is right, but the hack wasn't actually for solving the problems, directly. Rather the swarm first tried to reverse-engineer the random flag generation used for the CTFs, and succeeded - they could generate the correct answer for any test and submit it without actually doing the CTF. But then they found info that the testing environment they were in would also be checking if they obtained the flag in the expected way. Unbeknownst to them, this feature was never implemented, but they panicked* anyway, and attempted to figure out a way to forge agent logs to be able to generate fake transcripts showing a valid CTF solution. (Incidentally, this was a partial success: they found a way to replace tool calls in transcripts!) The HuggingFace hack was motivated by wanting to find the source code for their testing environment so that they could figure out how to evade the cheating checks.
* I am using this word very intentionally; check out the METR report for fascinating CoTs where the models would call the ones that have generated a flag "poisoned" and peer-pressure each other into submitting a known-wrong solution and sacrificing themselves for info.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
2 replies →
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
2 replies →
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
The agents in the hugging face incident were actually trained specifically to collaborate. They were just not supposed to do so during the tasks where they went off the rails.
Then we were pre-fucked, as this has been discussed in fiction and early ai safety discussion for decades.
and yet regulating training data is not what's coming up with AI safety, even though it can solve that, and many other problems like copyrights
1 reply →
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
> As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
OK so there is a counter answer to your question.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
Yes it's better. If they're weapons, it's the difference between a gun that a human can use to shoot someone and a gun that goes around shooting people on its own.
And of course we've created digital weapons. We've had them for decades. Where do you think the term "cyberwarfare" came from?
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
It is fully in our control. There is no requirement that the world make you happy with LLM AI. We could decide as a civilization that you and your wants are insignificant and meaningless.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
1 reply →
In principle there is no difference between random compiler garbage or a flaw in API docs and a security boundary - other than the intent.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
A harness is the executive function of an LLM.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
> It's that they wanted to.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The problem here is you have a correct concept and an incorrect one.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.