Comment by forgotaccount3
1 day ago
We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
Question who's never relied on this: how do you test whether your multi-AZ setup can withstand an AZ dropping? It sounds like your third party apps make it kind of pointless to test anyway, but I'm curious!
Ops type here, generally you have to fail an AZ and see what happens. If you have Dev/Staging environment that is exactly like production with plenty of tests, you can use that. I've seen only one environment actually like that though.
Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.
Thans! So do you have a staging environment for everything, then cut off the network or kill the instances, Aurora DB, etc.? Simulate some load while its going on, and then check for data loss? Then if that works try it in prod?
It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.
2 replies →