Comment by alexpotato
13 hours ago
I was the head of a global SRE team at a top tier hedge fund.
We started doing a weekly meeting where whoever was on call would do a table top exercise of an outage that happened the prior week.
The idea was to have someone else be the simulated person on call while the SRE from last week's oncall would talk them through the symptoms, what happened and where to look.
The idea was to spread knowledge around how incidents looked, what tools were used, what could have been done differently etc.
This was largely inspired by the following quote:
"Drills are for working on the infrequent actions that lead to big outcomes. A good example is heaving the ball from half court in basketball when the game is close. You can't control who will have the ball in that situation but you want everyone on the team familiar with what to do and how to do it."
How often are there outages? Is there a significant outage every week to do simulations for? The problem with our organization is that outages are so rare that weekly we have nothing to discuss.
They were usually small outages that we caught before they got out of hand.
If you never have outages, then you need to start getting either creative with the "table top" exercises or you set up a test environment that is very prod like and have someone randomly turn off components aka chaos engineering.