Comment by Art9681
15 hours ago
No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
15 hours ago
No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
That’s simply not true. I use S3 (well GCS mostly) for data that I wouldn’t be upset if it’s lost. And I pay the zonal discount for it.
3 replies →
Yeah agreed - any ephemeral stuff I need is generally in DynamoDB - S3 (and database) are for permanent storage.
Single AZ S3 has other benefits. The point isn't the price, it's that it's _highly performant_ since you can keep all of your reads in the same AZ
It's a great service for large caches. For example, we process a lot of imagery that we download from third-party providers. We save a lot of latency by storing the data in a single-AZ S3.
If it dies, we will just have to re-download the data.
> EBS data data (sic) is unrecoverable, and that's what's reported.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
> The article does not tell us what products were impacted.
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
> to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
I had a similar discussion in the low 1990s at a large electronic manufacturer that had a pair of Unisys mainframes, one in São Paulo and another in Manaus, in the Amazon region communicating over a satellite link (two large 3m dishes on both places). There was one question of what happens if both fail at the same time, and I pointed that anything that takes out São Paulo and Manaus at the same time is a civilisation ending event, and we shouldn’t worry too much about that.
Anyway, we had a load shedding agreement with a large bank across the street from São Paulo and we could switch over to their Unisys mainframe in a matter of minutes, and vice versa.
You also need to understand whether the generator backup actually runs everything. Where I work it doesn't. Only "essential" systems get backup power.
And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.
That's probably already true for 7 days or less
3 replies →
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
"Daddy, where were you when the flames reached our house?"
"I was in the office, reviewing Terraform plans"
Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries
A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.
Do you think they could ask for a backup
Typically 300 miles geographically but could be hard in some Gulf States
In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all.
Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.
5 replies →
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
Nothing is ever real-time. Eventual consistency leads to some data are not backed up.
You talking as if this is some mom-and-pop shop that you run.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
Did local laws permit?
You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.
[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
Not all data is allowed to leave all countries.
what a silly notion.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
Isn't S3 claiming eleven nines of data durability?
https://aws.amazon.com/s3/storage-classes/
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
1/f noise strikes again