Comment by packetslave

5 years ago

Back in Ye Old Dark Ages, I caused a BIG Google outage by running a routine maintenance script that had been run dozens if not hundreds of times before.

Turns out the underlying network software had a race condition that would ONLY be hit if the script ran at the exact same time as some automated monitoring tools polled the box.

At FAANG scale, "one in a million" happens a lot more often than you'd think.

> At FAANG scale, "one in a million" happens a lot more often than you'd think.

And it happens less than you think too, sometimes, which I think is closer to the original point.