Comment by iainmerrick

It’s a good cautionary tale -- in hindsight the danger signs are clear, but it’s also clear why you thought it was OK and how third parties unfortunately let you down.

The “agent’s confession” is the least interesting and useful part of the whole saga. Nothing there helps to explain why the disaster happened or what kind of prompting might help avoid it.

The key mistake is accidentally giving the agent the API key, and the key letdown is the lack of capability scoping or backups in the service.

The main lessons I take are “don’t give LLMs the keys to prod” and “keep backups”. Oh, and “even if you think your setup is safe, double-check it!”