Comment by ddalex
5 years ago
Not the original googler responding, but I have never experienced what they describe.
Postmortems are always blameless in the sense that "Somebody fat fingered it" is not an acceptable explanation for the causes of an incident - the possibility to fat finger it in the first place must be identified and eliminated.
Opinions are my own, as always
> Not the original googler responding, but I have never experienced what they describe.
I have also never experienced this outside of this single instance. It was bizarre, but tried to reinforce the point that something needed to change -- it was the latest in a string of major customer-facing outages across various parts of TI, potentially pointing to cultural issues with how we build things.
(And that's not wrong, there are plenty of internal memes about the focus on building new systems and rewarding complexity, while not emphasizing maintainability.)
Usually mandatory trainings are things like "how to avoid being sued" or "how to avoid leaking confidential information". Not "you need to follow these rules or else all of Cloud burns down; look, we're already hemorrhaging customer goodwill."
As I said, there was significant scar tissue associated with this event, probably caused in large part by the initial reaction by leadership.