Comment by solatic
1 day ago
Author has a good head on their shoulders, but few if any companies are going to spend time on incident simulations for their SREs.
Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or... It is the least-sexy operations work that exists. No executive cares about this. Ops organizations push for flashy work, same as everybody else: new infrastructure for new projects, cool chatbots, new flashy dashboards, make charts go up and to the right, etc.
Airline pilots go through disaster simulation training because the government mandates that training. If it wasn't a condition of holding a pilot's license, no company would pay for it.
Want SREs to spend time training for disasters? Take a step back. Support professional licensure. Make it a condition of holding a license. You won't get industry-wide professional behavior until you professionalize the work. It won't happen without licensing because every corner cut that is not immediately visible to consumers translates to additional profit, and increasing competition eventually requires these corners to be cut in order to keep up with competition and stay in business. Forcing all players to submit to licensing requires all players to pay these costs and thus forbids them from cutting them to become more competitive.
I was the head of a global SRE team at a top tier hedge fund.
We started doing a weekly meeting where whoever was on call would do a table top exercise of an outage that happened the prior week.
The idea was to have someone else be the simulated person on call while the SRE from last week's oncall would talk them through the symptoms, what happened and where to look.
The idea was to spread knowledge around how incidents looked, what tools were used, what could have been done differently etc.
This was largely inspired by the following quote:
"Drills are for working on the infrequent actions that lead to big outcomes. A good example is heaving the ball from half court in basketball when the game is close. You can't control who will have the ball in that situation but you want everyone on the team familiar with what to do and how to do it."
How often are there outages? Is there a significant outage every week to do simulations for? The problem with our organization is that outages are so rare that weekly we have nothing to discuss.
They were usually small outages that we caught before they got out of hand.
If you never have outages, then you need to start getting either creative with the "table top" exercises or you set up a test environment that is very prod like and have someone randomly turn off components aka chaos engineering.
Hmm that’s strange, in my experience it is the other way around - Claude is super diligent with infra and will _insist_ on double checking and trying everything for real before committing.
When I was doing this myself I would read the docs and just implement them - claud is going about doing real software archeology to figure if what is said is actually the truth or it’s stale/inaccurate/buggy.
I’ve become 10 times more diligent because it is a lot easier to do. It’s no longer Urgh it’s good enough let’s ship it, now it’s “sure put a leg on it to figure it out and double check it”.
Backups are _tested regularly_ now because LLMs make it cheap to do so.
The only problem is when new engineers who haven’t learned these things Pre-ai now don’t really get why it is needed in the first place and will often lead the agent astray.
I think to address this we need to change or improve our training routines in general for humans. I think a lot of companies nowadays just skip that and deploy a company wide skill/policy for the agents, but don’t transfer the underlying skills to the devs themselves.
> Backups are _tested regularly_ now because LLMs make it cheap to do so.
sighs heavily in 90's sysadmin
Testing backups is not just a question of whether or not the restore command works. Go back and read the Tao of Backup: http://www.taobackup.com/history.html . The application itself (in its current version, with its current features) needs to work with the backed-up data, and the only way to verify this is to attempt to actually work with the data.
If you don't trust your agent to ship to production without manually reviewing the output (in some way), you have no business trusting your agent managing your backups. The agent writing some tests doesn't mean that the tests adequately handle all of your actual scenarios, let alone that your system will adequately handle data that is missing since the last backup.
Good news: Eventually - and fairly regularly! - the agent will fuck up so badly you have to test the process!
>Why not? Because even pre-AI, very few companies spend time practicing restoring their backups, or disaster recovery, or picking infrequently-used runbooks to practice, or seeing whether they can easily rotate secrets without downtime, or trying to redploy the system onto another vendor's cloud/platform, or, or, or...
Exact. Let insurance cover it, say sorry to your customers twice and shwoop never happened.