Comment by lukeschlather
5 days ago
My read of the situation is not only did they say that, they had countermeasures (a watchdog agent) which stopped the agent and said "what are you doing, stop that" if it tried to download the solutions from Github. But the agent figured out Hugging Face had another copy of the solutions and it figured out a way to go after the solutions without tripping the watchdog.
Although maybe they didn't have a watchdog agent; I definitely think using watchdogs like how Claude wraps pretty much every single tool call in Haiku to check if the command is reasonable is very necessary for unsupervised work. I suspect in the future you'll want Fable-class models wrapping every tool call, possibly with several checks "is this consistent with the goal? does it do anything unreasonably dangerous in pursuit of the goal?"
They didn't have watchdog agents - those exist for their production models but had been deliberately removed for the purpose of this evaluation.
OpenAI wrote about how their mechanism for that in production works here: https://openai.com/index/safety-alignment-long-horizon-model...
> We created a monitoring system that reviews the model’s evolving trajectory for signs that it is bypassing a user constraint or safety boundary. The monitor observes not just a single action but the entire trajectory.
So core ChatGPT models are all abliterated?
Open weights releases should have a companion model that censors boobs and Tiananmen and then we would have a useful model and something we could fine–tune into a useful model instead of a half–useful model needing a lobotomy.
And I mention Tiananmen because I feel like the Western models are built easier to abliterate judging by the quality difference.