Comment by NitpickLawyer

I think at some point the model itself is asked if the command is dangerous, and can decide it's not and bypass some restrictions.

In any case, any blacklist guardrails will fail at some point, because RL seems to make the models very good at finding alternative ways to do what they think they need to do (i.e. if they are blocked, they'll often pipe cat stuff to a bash script and run that). The only sane way to protect for this is to run it in a container / vm.