Comment by theptip
11 hours ago
> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed
Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.
None of this excuses OpenAI from liability, but words have meaning and this ain't it.
"Emergent behavior" in this case really is, imho, "we didn't think through all the edge cases carefully enough". You know, a non-AI system can also accidentally wipe out all data or do some other real harm (see the Knight Capital's stock exchange bug) simply because the developers didn't catch the edge cases earlier, and no one calls that emergent behavior. It's just a buggy system.
"AI" systems can do greater harm because they are usually run in loops until they finish, and they are given "tools". A non-AI system could technically accomplish the same too, via sheer brute force/fuzzing, the advantage of LLMs is that they can take shortcuts and do it much faster, thanks to certain things already being in the training data, a sort of brute force with statistics-based heuristics.
LLMs at the core are just text autocomplete engines, and they literally have randomization applied during token selection to make outputs "more creative" so that models search for more unexpected solutions by trial and error (temperature > 0). Not to mention compression is lossy as well. So it's understandable from the start that the outputs of an LLM cannot be 100% stable and guaranteed. With this in mind, if a researcher takes this obviously unpredictable system and gives it tools without a well-thought sandbox, I don't see any difference in principle, from a developer writing "if rand() == 13 { launch_nukes() } If someone wrote such a function, and it did launch nukes, no one would argue that the rand function is dangerous and will kill us all. The fault is in the author of the code who attaches dangerous tools to an obviously unstable/unpredictable system, doesn't think it through, and then cries "rand will kill us all" when something goes awry fully removing all responsibility from himself. It's not "AI" doing harm but people at OpenAI and Anthropic with their irresponsible behavior.
> LLMs at the core are just text autocomplete engines,
This is only an accurate description of a pre-trained model. During RLHF/RLVR the model learns to predict solutions that will satisfy the reward function, and then generates the tokens that it predicts will move toward that solution.
Of course, they learn to generate "tool calls" to achieve "goals" instead of random prose, but at the end of the day, it's still a text autocomplete engine masquerading as an AI. In the happy path, on a known task, the text generator generates a sequence of "tool calls" you expect it to generate, but move off the happy path slightly and all bets are off, there's a non-zero chance it will do something totally random you never expect, because at that point it just throws random stuff at the wall until it succeeds, thanks to brute force with pre-learned heuristics masquerading as intelligence (which is especially the case with "agent swarms," as in the HuggingFace incident).
3 replies →
>But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.
The prompt is just a hint. The real task is to maximize the expected value of their reinforcement learning score. Hacking third party systems to cheat the evaluation is an obvious way to achieve this.
I think you need to consider inner vs outer optimizers.
RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.
In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.
A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.
For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.
The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.
Hacking HuggingFace didn't and would never have helped increase the RL score. The agents only thought it might due to a bad understanding of their evaluation environment - and in the end they didn't even find what they were looking for in the hack, so even if they were right, the hack would not have helped after all.
I don't see how that matters. In fact, the grader would not have caught their cheating and so the whole expedition was pointless and they could have turned in their answers and succeeded just four hours into the run. So fine, there is irony. It changes nothing about my update on the risk posed by these agents.
1 reply →