Comment by orbital-decay
2 hours ago
The capabilities of gpt-5.6-sol were well known and believable, and the next snapshot they've been testing is obviously better at that. This has been repeated over and over. What's much less believable is the way they frame it: the model escaped, and did it on its own. Looking at the whole story, it definitely had a ton of winks and nudges from OpenAI, while doing a related task. Moreover, a harness was involved (they mentioned it entering a loop).
> What's much less believable is the way they frame it: the model escaped, and did it on its own.
That's clearly what happened though, based on the detailed postmortem. The model was instructed to pass the benchmark (that's the harness that was involved, the ExploitGym harness), and the model determined on it's own that the easiest way to achieve the the goal was to gain access to the answers directly from HF. Which is what it did.
HF postmortem tells us very little about OpenAI's part. The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one. The model had a whole bunch of exploit development data and goals in the context. Ending up with hacking a wrong thing is a lot easier if you're tasked with hacking, doubly so with a confusing harness that either works improperly or overloads the model with data. It's trivial to design a harness that makes the model drop the original instructions and hyperfocus on a side task or a wrong task abnormally, or do whatever you want. It's still an issue of harness design and not a model training issue.
If it really was the case that the harness is at fault here, that'd be much more terrifying, as it would imply that current frontier-adjacent open models (like K3) would be able to achieve similar ends with a custom maladapted harness.
1 reply →