Comment by bbor

Putting aside the 100% baseless, unfalsifiable accusations of intentional accidents for a moment, I hope we can agree on one thing: Irregular either needs to hire us or go out of business cause seriously it's beyond parody at this point. WTF is going on over there? There's surely dozens of firms chomping at the bit for these contracts already, and the field hasn't been around long enough for them to build some sort of unique expertise moat that would justify this many public failures.

Basic sandboxing is not exactly rocket science after all,[1] and it sure seems like they're missing a whole stack of swiss cheese slices on top of that. Some basic precautions off the top of my head that seem very likely to have caught all of these incidents:

1. Alerts based on telemetry (most importantly, HTTP requests), both explicit (normal) and semilatent (use DL to confirm an intentionally-eager alert before firing it).

2. Latent alerts based on transcripts, e.g. noticing when a thousand agents start mentioning a secret off-premises hangout spot. Even mere embedding comparisons seem likely to catch such a blatantly misaligned sentiment as that one, especially with n>1000.[2]

3. Pausing agents completely until an on-call engineer can rule on ambigious situations or potential issues -- surely security is worth <$1 in lost token cache, especially for a security company?

4. Superheavy orchestrator/baby-sitter models checking in on cybersecurity eval transcripts periodically just in case -- again, would be a neglible cost. Could also be made available to the agent as the first line of defense for clarifing a rule ad-hoc, feeding even confident responses to a queue that is reviewed asynchronously by humans within a workday.

5. Or, hell: just clearer prompts? I'm a cybersecurity noob, but I still feel confident we can write really productive, challenging CTFs without leaving questions open like "maybe I'm supposed to hack my own harness?"

Seeing as they haven't been fired by any of the big 3 yet, they're presumably smart, experienced, dedicated folks. And I'm not normally a "if only I were in charge!" person, I promise. But c'mon.

Perhaps I'm missing something?

[1]: To their credit we have gotten tidbits that indicate some blocklists & such exist, e.g. the German wiki hacks had to work around a blanket ban of POST requests.

[2]: This hints at their insane decision in one or both of the OpenAI incidents to just bandaid up the issue when found, which supersedes all of the above. You can stack swiss cheese slices a mile high and they'll still fail to protect you if the attacker gets to keep retrying & adapting indefinitely.