Comment by SpicyLemonZest

I think it's pretty clearly negligent to set up a harness that will run whatever outputs it receives from your LLM if you cannot reliably stop it from outputting hacking instructions. I've said in the past that I'm sympathetic to the idea of testing your anti-hacking controls, but that doesn't seem to be where this incident came from.