Comment by RamblingCTO
4 hours ago
I've been working on that for a client since yesterday (as a fractional CTO). Pretty hectic, basically nothing really works and we don't know yet if all data is lost or if anything is recoverable or when AWS UAE will become functional again so we can recover that region.
Finally, I have a very good argument for multi-region deployments ;))
that's my go to website atm: https://health.aws.amazon.com/health/status
What do you mean 'finally' - surely 'redundancy' or 'natural disaster' is reason enough.
To a lot of managers/startup execs this is something like "we won't ever need this". And I'd agree to some extent for not so important/rebuildable services. Depends on what you need. In startups, you don't have infinite time you can spend on stuff. But this makes a good case: if a geographical region only has one AWS region, don't keep data or run services that can't be easily rebuild somewhere else. In europe you can just pick two AWS regions and you stay in the same regions, UAE not so much.
I don't think this changes anything. It is always the same argument for me.
"How often do those happen?"
That was my point: now we have a very specific case we can argue. I always used the "what if a plane crashes into the data center" argument. Now I can say: in one of my engagements, we lost a datacenter completely because of a drone attack. That's a first for AWS as well, but we can draw from reality, not hypotheticals.
Severity: Disrupted
So if data won't be recoverable you all will mark it something like "Status: FUBAR" or some equivalent term?
FUBAR, I would vote for that
I wonder if ‘Apocalypse’ or ‘Molten Slag’ would be considered professional enough.
I'd accept "X_X" as status.
We didn't do multi-region deployments, but we did store database backups in a separate region just in case something really bad happened and our AWS region became unavailable. Also had a plan/some ready Terraform stuff in order to start setting up a deployment if it became apparent that the region wasn't coming back anytime soon.
IMO, if you're using AWS and not replicating your data somewhere else, this should be an eye-opener for you.
Not sure why everyone read this as me doing anything here, I'm a fractional CTO, which is kind of an advisor. Nothing invaluable will be lost tho. It's not the core platform, just a localized version for specific customers in the region.
You don't already have a DR plan in place?
Of course, no worries. Nothing irreplaceable will be lost ;) It was meant as a general example to be used in future arguments for everybody in tech leadership.
Any backups?
> Any backups?
Yup something has to be said too about good old offline backups on fat tapes.