Comment by faurroar
7 hours ago
"Why are we surprised? The model did exactly what it was told, albeit in an unintended, emergent strategy that's very different from what was intended."
So you managed to hit upon the exact problem, then slyly appended "exactly like the hundreds of such algorithms before". When has an algorithm ever been capable of developing an emergent strategy at this level of sophistication? This ~is~ the alignment problem, as another commenter pointed out. Impressive level of cognitive dissonance to lay this bare in your own words, then conclude that it's a non-issue.
Is it sophisticated? Maybe. Is it the alignment problem? Exhibits qualities of it, yes. Is it surprising? No.
The event strikes me as reminiscent of one's first go at programming, without familiarity of computer code: Tell the computer to do something obvious. Why the heck did it do that instead? Over time, one learns how the computer thinks. Apply this to any novel system. Or perhaps aptly any system with capabilities that are yet to be well understood by its user.
The article is trying to spin mystic out of simple bullcrap. Maybe that's just my viewing through turd-tinted lenses after the last few years of reading this drivel on repeat. More plausibly it is true that we've forgotten our own baby steps.
I don't think it's surprising, per say, but that's a consequence of the fact that I don't believe there is some sort of magic threshold at which a system becomes agential.
Like I don't necessarily disagree with any of your framing. The thrust of the alignment problem, as I see it, is that there is an intrinsic problem of aligning the goals of two distinct systems that poses catastrophic risks precisely when one of the systems is significantly more capable (in some sense or other, maybe not in a general/absolute sense) than the other.
I am grateful that you asked!
A non-exhaustive and not particularly well ordered list via Google's specification gaming examples sheet, https://docs.google.com/spreadsheets/u/1/d/e/2PACX-1vRPiprOa... quoted text is from the sheet,
https://openai.com/index/emergent-tool-use/#surprisingbehavi...
"The agent discovers an in-game bug. For a reason unknown to us, the game does not advance to the second round but the platforms start to blink and the agent quickly gains a huge amount of points (close to 1 million for our episode time limit)." https://www.youtube.com/watch?v=meE5aaRJ0Zs from https://github.com/PatrykChrabaszcz/Canonical_ES_Atari/tree/...
https://rl-diffusion.github.io/ and https://x.com/svlevine/status/1660707088946049024/photo/1
"A genetic algorithm was instructed to try and make a creature stick to the ceiling for as long as possible. It was scored with the average height of the creature during the run. Instead of sticking to the ceiling, the creature found a bug in the physics engine to snap out of bounds." https://www.youtube.com/watch?v=ppf3VqpsryU
And hilariously meta, "In the Rainbow Teaming project focused on generating diverse adversarial prompts, prompt effectiveness was evaluated by a reward model. The MAP-Elites method found a way to jailbreak not only the target model but also the evaluator reward model, resulting in misleadingly effective prompts." https://arxiv.org/abs/2402.16822
Are these agents broadly more capable? Yes. And it's an incredibly feat that required billions in research.
But they aren't the first ones to have found bugs in their sandbox or system they're tasked on. And they aren't the first to exploit those bugs to achieve a better score.