Comment by Kim_Bruning

So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P

https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)

It's not a pure security issue.

If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.

Change the design, and your agents might not see a need to leave in the first place.

Look at the chain:

Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.

Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :

* A cooperative grader.

* A way to report "this is impossible" which still gets scored.

* A better explanation of the long term purpose of the task.

Even just these three provides offramps or mitigations that can reduce the pressure to cheat.