This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
Well, listen, you do know that even when the largest most ambitious and most sophisticated things are shipped, each owner of each specific part (could be many many owners) basically , to the best of their ability, prayed that nothing particularly bad happens when it’s shipped off. Truly, that’s the best a mortal human can do, pray their part doesn’t break.
So then that big thing comes to you. It’s all kind of … held together by a prayer …
Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.
It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life.
For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.
Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are shared across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't share across data centers at all, only across racks within a data center.)
Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.
Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).
But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 had durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!
The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.
The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
Yes, this, and the article seems pretty clear that only "some data stored exclusively in Bahrain" is affected. Add to that the option to store data with reduced redundancy (in S3, for example), and I don't really see what the drama is about.
Amazon only claims "99.999999999% durability" per year, even for the properly replicated stuff.[0]
Clearly since the MBAs took over AWS standards are not anymore what they used to be.
That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment.
He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions...
AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list.
And their Well Architected Framework and other docs make this plenty clear:
"It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)."
Also...
"All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region."
And also for single-Region / Multi-AZ architectures:
"Where possible, you should also copy data backups to another AWS Region as an additional layer of protection."
"AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters."
This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.
If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract.
Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.
Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead.
Now, there are a few things to consider:
- AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data
- You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data
When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.
> "The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand," AWS said in the status update
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop insisting on where the data resided for jurisdictional purposes and start talking about where administrative access and encryption keys lived.
[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!
This is a very engineer-centric view. I studied economics in school, so an analogy in that realm is ironically how all countries should specialize and raise the PPC curve. The reality of the situation was that in 2010 not many people understood how powerful big data actually was. Data sovereignty is actually quite logical when you consider the scale and power of not only the company, but the US as a whole. I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws. You can also create reasonably diversified data silos within a country.
As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.
I mean, it was clear and open that USA will spy on any data stored in there. Because foreigner do not get legal protections.
And second, the USA is in the middle of power grab that completely ensures any data stored there will be taken hostage wherever suitable for "negotiations".
and start talking about where administrative access and encryption keys lived
Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.
... in the UAE, so only less risky if your office is remote enough it doesn't also make a nice target (like, say, located within/near Dubai's Internet City).
My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical report of why they can't recover things.
11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.
The footnote which says that is the design durability against equipment failure literally begins:
> In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
> US refuses payout for soldiers who died in Iran 'because it isn't a war'
> In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.
If a senior leader at Doordash swore to God that your sandwich would 100% guaranteed make it to you, regardless of whether or not there was a car crash, then yeah, maybe you should complain
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system.
At some point you need to start worrying about asteroid defense too.
Hot take: if you have a service that is 11 nines reliable, but there is an underlying component whose reliability is lower, cap the nines to that component.
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
> EBS data data (sic) is unrecoverable, and that's what's reported.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
> The article does not tell us what products were impacted.
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
> to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.
[1] That's a quote, including the Boston accent.
[2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat)
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat)
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
The data center layout should look something like this:
me (Middle East)
├── me-south-1 (Bahrain) DOWN since 2026-04
│ ├── mes1-az1 DOWN
│ │ └── mes1-mct1-az1 (Oman, Muscat) ???
│ ├── mes1-az2 DOWN since 2026-03-01
│ └── mes1-az3 DOWN
├── me-central-1 (United Arab Emirates)
│ ├── mec1-az1
│ ├── mec1-az2 DOWN since 2026-03-01
│ └── mec1-az3 DOWN since 2026-03-01
└── il-central-1 (Israel, Tel Aviv)
├── ilc1-az1
├── ilc1-az2
└── ilc1-az3
Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.
If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".
So many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.
For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably
a) letting customers in other areas know that their data is backed up to another continent
b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"
For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.
Ahhh the downside of “data sovereignty” rules and the de-globalization meme strikes again. I’d bet a bajillion dollars the reason this happened is due to government thinking it’s a good idea to make it illegal to store data outside their country.
Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi-region residency…as long as the European Commission doesn’t stupidly keep trying to make encryption illegal too!
Hint: Russia absolutely knows where your data centers are.
Yes there’s never been division or wars in Europe and the EU will always exist…
Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy.
I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left.
And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.
It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.
The wording certainly seems very careful:
"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
Israel had been trying to get every president to bomb Iran for decades. There's a reason they wouldn't do it themselves. They finally got a president stupid enough to listen.
This is not exactly a nuanced view of the conflict, and in either case, the fact that you don't like that someone on the other side of the world is chanting death to America doesn't give you a bonus card for a free attack.
Seriously, it's like people, when deciding whether to launch a war or not, are not thinking "how will the other side react and will this conflict benefit me" but instead they are only thinking "does this nation deserve to get hit".
Well, news flash, your moral outrage does not translate into you not suffering more than your opponent during a conflict. It's a completely separate issue, and a personal issue between you and your priest or rabbi. When it comes to starting wars, you have to look at military capabilities and long term outcomes, not "does this nation deserve to be attacked".
> This is the first administration in a long time with the cohones to do something about it.
Containment worked WAY WAY better than "doing something about it" and surrendering the strait to them. They "solved it" by aerosolizing asbestos everywhere and still have no cleanup plan. Sometimes containment works much better, which is why intelligent foreign policy worked the problem from that angle.
Wasn't victory declared 1 year ago and also a number of months ago?
Do you know why Iranians were chanting "Death To America"? Because we toppled their democratically elected government in 1953 and installed a brutal dictator so that we could continue to pillage their oil.(1) When the mullahs finally toppled that dictator in 1979 they had reason to be angry with us. This is especially true because almost immediately afterwards, in the 1980s, we used our proxy Saddam Hussein to launch a war against Iran, in which over a million Iranians were killed. During this time we provided Hussein with the means to make both chemical weapons and biological weapons (2), which Iraq used extensively against Iran. I suspect if another country did the same to us, we'd be chanting "Death to XYZ" too.
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
I'm comfortable enough with the Sydney and Melbourne AWS regions - about 700km (400 miles) apart and with (at least) 3 AZs in each. If something takes out enough AWS datacenters to lose some of work's or client data stored across all that, the uptime and resilience of the CRUD platforms I'm responsible for will not be very high on my personal priority list. (At least on AWS datacenter is within 10km of my home. I'm hoping that well before Australia gets involved in the sort of geopolitical conflict that might mean missile strikes against civilian infrastructure, I'll have headed bush to hang out with my off grid friends)
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)
We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
This interview with an AWS leader isn’t aging well, from CBS Sunday morning:
Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"
https://www.cbsnews.com/news/cloud-computing-loudoun-county-...
That is actually surprising to me. Claims like that are pretty common, they make sense and they should be true, so even though I don't really know AWS (/Backblaze/Azure/whatever) redundancy planning in enough detail, I used to trust them. It's really worrying when they outright say it will be ok, and then a week later it turns out to be not ok.
Well, listen, you do know that even when the largest most ambitious and most sophisticated things are shipped, each owner of each specific part (could be many many owners) basically , to the best of their ability, prayed that nothing particularly bad happens when it’s shipped off. Truly, that’s the best a mortal human can do, pray their part doesn’t break.
So then that big thing comes to you. It’s all kind of … held together by a prayer …
Trust me I’ve worked at these big places. You wouldn’t believe how much fucking luck and grace from God is allowing you to do anything with your digital life. It’s a mindfuck of a tangled mess out there, eternities worth of written code that only God ensures works together at this point, only to get more hidden with AI.
It really is important to understand the failure modes that the durability model accounts for and what it doesn't. It only accounts for "normal" failures, like an HDD reaching end of life.
For example, you mention Backblaze. Backblaze has public posts about their durability model. They claim to use 17:20 Reed-Solomon erasure encoding. That means there are 20 shards of a blob, and you can lose 3 of them and still reconstruct the blob.
Think about that for a second. If they store 4 shards in a datacenter, that means that a loss of that one datacenter is sufficient to lose the blob, forever. That entails that blobs are shared across a minimum of 7 data centers, or the loss of one data center might mean permanent data loss. Which one do you think is true? (In fact it's pretty clear from Backblaze's public posts that they don't share across data centers at all, only across racks within a data center.)
Now, AWS's availability guarantee — not their durability guarantee — entails that they use a less cost-effective erasure coding ratio. S3 is designed so that your blob is available even if a whole AZ goes down, and it's well known that most AWS regions have only 3 AZs. Therefore, if you tolerate the same number of shards lost to HDD failure as Backblaze in your durability model (3), then you might need 17:30 erasure coding to get the same durability and the required availability. That means S3 is storing way more physical bytes than Backblaze — 1.76x the logical size of the blob, instead of Backblaze's 1.18x. That's more expensive, but it also gives you better availability.
Which is also why One Zone S3 is cheaper — if you don't care about the availability guarantee, S3 can do what Backblaze does and save 33% on physical bytes, and they pass on 40% of those savings to the customer (this is fairer than it sounds — there's more overhead than physical storage bytes).
But here's the thing. AWS has more redundancy built in than Backblaze because they make availability guarantees in addition to durability guarantees. BUT the durability model is the same, which is why Backblaze can claim equivalent durability to S3. S3 in fact has better durability — they can survive the permanent loss of an AZ without necessarily losing blobs stored there (with the exception of One Zone blobs), and Backblaze cannot. But that's not actually a factor of the durability model, which is just taking into account normal events like HDD failure. Instead, S3 had durability that's more resilient to AZ loss because of their availability model. It's a side effect that isn't actually part of the durability promise!
The devil is always in the details. Somehow I feel that when we offload the responsibility to some one else we get this feeling that the other person/entity would be doing full diligence and whatever else is required to carry out the job perfectly. However in reality most of the times they just do the bare minimum to pass your evaluation criteria to get the job.
7 replies →
The caveat is always "if you're using the service correctly" which is not necessarily free. Meaning taking advantage of multiple geo zones, building in redundancy to your stack, etc. Like everything he said is possible if your technology stack living in AWS was designed to survive it. Everyone who has ever had the "we lost your data" email from AWS knows at the end of the day the cloud is just someone else's data center with neat provisioning tools and services.
47 replies →
It's probably a dedicated government type thing where the data is housed seperately from normal AWS
“It’s not the cloud. It’s just someone else’s computer.” - MM
Isn’t the problem that multiple datacenters in one zone were blown up?
Yes, this, and the article seems pretty clear that only "some data stored exclusively in Bahrain" is affected. Add to that the option to store data with reduced redundancy (in S3, for example), and I don't really see what the drama is about.
Amazon only claims "99.999999999% durability" per year, even for the properly replicated stuff.[0]
[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
1 reply →
Multiple zones impacted.
They had nine years and more money than god to build redundancy in an unstable region.
14 replies →
articleAuthorId: 30dacdbc-6a8a-11e2-9d12-0018fe8a00b0
articleAuthorName: cbsnews (hidden byline)
articleSecondaryAuthors: n/a
articleEditors: n/a
Is the author a human or machine? Google shows 1 result for "30dacdbc-6a8a-11e2-9d12-0018fe8a00b0" and Brave Search shows 5 results.
Clearly since the MBAs took over AWS standards are not anymore what they used to be. That marketing guy should not be talking to the press, as he does not have the skills, and if somebody happens to say...our data center we wont lose any data if we have an issue, without qualifying it will depend on what quality of service, and usage of our services you setup ...is the type of technical answer that should make a hiring interview stop at the moment.
He is also violating an enormous amount of compliance requirements, by disclosing the location of the data center, and having strange people inside making a tour. Did he vet the crew and their accompanying party? Did one of them accidentally left some kind of device within the insider perimeter? There at least one or two ISO certifications he is violating there. As customer I would be asking questions...
AWS always made very clear they wont copy your data to another region as only you know what your compliance and data residency requirements are. But at the same time they always said, its up to you to come your with your disaster recovery strategy based on your project requirements. And it has always been the case copying your critical data to another region is one of the first things on your check list.
And their Well Architected Framework and other docs make this plenty clear:
"It is a good practice to always make backups of your data, and copy these to another site (such as another AWS Region)."
Also...
"All DR strategies require that data sources are backed up within the AWS Region, and then those backups are copied to the recovery Region."
And also for single-Region / Multi-AZ architectures:
"Where possible, you should also copy data backups to another AWS Region as an additional layer of protection."
"AWS Architecture Blog — Disaster Recovery Architecture on AWS, Part II" has a whole section named "Backup to another AWS Region": "By copying your data to another Region, you can handle the largest scope of disasters."
https://aws.amazon.com/blogs/architecture/disaster-recovery-...
Or "Creating backup copies across AWS Regions" - https://docs.aws.amazon.com/aws-backup/latest/devguide/cross...
This whole thread of people literally saying , "on no I trusted them...I did not know they could lose my data", with no technical context...is the the kind of incompetence I would expect from a generation raised on vibe coding and llm prompt driven miseducation...
There's something reassuring in this for me.
There's a lot of magic & handwaving from hyperscalers like AWS about redundancy. I always wondered about some of the engineering to make this absolutely (and literally) bullet proof. At the end of the day most of their answers when you push hard enough involved paying 2-3x to run everything across multiple zones/regions, and lots of awareness in your application to handle this.
In any case, I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
> I think it's good that when a data center blows up the data is lost. Noteworthy for future skynet situation, etc.
Doesn’t really apply, because the only reason data was lost is because customers chose not to replicate it to other regions, either because of legal data residency requirements, cost, or just not bothering.
If Skynet wants to make sure it’s backed up, none of that prevents it from doing so. Although it would be amusing if Skynet was stopped by a billing alert when it tries to copy itself to another region.
3 replies →
that quote is definitely making its way into a lawsuit
Yes. It's super hard to answer interviews or customer demos. Every sentence you say should include all preconditions that were said in previous answers, within the same context, because you may be quoted. You should think in live about all possible contexts in which your app might be used, and your speech should be as detailed as a contract.
Obviously here, he should have mentionned that they can recover a hit on a single data center, provided the customer chose multi-AZ hosting. That's probably why companies run their ads on "This watch is a legacy for your children" rather than any material claim.
Unless you have a synchronous like setup where you don’t acknowledge data writes unless the remote has aconowledged them first, you will lose data in case your datacenter is hit by a warhead.
Now, there are a few things to consider:
- AWS best practices recommend multiple AZs for workloads and cross-region backups for things like databases and other “stateful” data
- You have to read the fine-print on what AWS offers in terms of recovery: do they reffer to their own infrastructure when they say “you won”t notice” or your data
When Google’s Paris colocation facility was flooded and all AZs there went dark, they sent an email saying “restore from backup in another region and if we can restore your data, we will make it availbale to you”. They did not even issue credits for the downtime.
He forgot to add "if it's Multi-AZ" ;)
> "The damage to our infrastructure spanned multiple Availability Zones and exceeded what our regional and multi-AZ services are designed to withstand," AWS said in the status update
The me-south-1 (Bahrain) Region was three AZs: https://docs.aws.amazon.com/global-infrastructure/latest/reg...
multi-AZ doesn’t help against multi-AZ drones :)
9 replies →
But AI chatbot advised him that was the case!
I wonder what kind of "some data" is: military installations in Arab states and Israel?
I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.
Hi, I'm from the future. You might want to consider storing the data somewhere besides an Azure datacenter in the UAE.
> the government requirements requires me to store data only in UAE!
4 replies →
Are you me from the future? I'm not talking to future strangers. Tell me to deliver the bad news myself from the future, in the future.
1 reply →
I'm from the past and I concur.
Hi, I'm from the past. When countries in the 2010s -- especially Western countries -- started seeing data residency requirements as an acceptable aspect of national policies, as opposed to a weird authoritarian thing that only China and Russia imposed on their citizens, we[1] spent a bunch of time explaining to their lawmakers that having geographical redundancy was a good thing, actually, and that you should stop insisting on where the data resided for jurisdictional purposes and start talking about where administrative access and encryption keys lived.
[1] OK, "we" here is probably just me -- it was one of those things where the chances of successfully convincing anyone was so small, and the commercial advantages of just nodding along, and then changing your product offering was so great, that really very few people raised it or had reason to. But somebody had to!
This is a very engineer-centric view. I studied economics in school, so an analogy in that realm is ironically how all countries should specialize and raise the PPC curve. The reality of the situation was that in 2010 not many people understood how powerful big data actually was. Data sovereignty is actually quite logical when you consider the scale and power of not only the company, but the US as a whole. I can assure you that lawmakers were not thinking about efficient disaster recovery plans or back ups when they made the laws. You can also create reasonably diversified data silos within a country.
As an aside, it is quite crazy the world we live in. I am with the majority where I expected Amazon to be more redundant, but I still marvel at the assumption that a US dev can spin up multiple redundant and data sovereign servers in dozens of countries with efficient caching, failover and redundancy (enough to survive an earthquake or targeted missile attack) from their own home. Even a few hours of outage in a foreign country is considered unacceptable.
5 replies →
Is there not more than one data center in your country? Is the power feed at your office too small to put a computer there?
3 replies →
I mean, it was clear and open that USA will spy on any data stored in there. Because foreigner do not get legal protections.
And second, the USA is in the middle of power grab that completely ensures any data stored there will be taken hostage wherever suitable for "negotiations".
and start talking about where administrative access and encryption keys lived
Yeah, that was/is just another problem. Considering how that was actually handled in the real world before data residency laws came into force, I'm glad 'we' didn't convince those countries to put their citizens data at risk.
9 replies →
you know this is why china and russia were stealing data, so that they can provide you backups if you lost yours /s
Check if the requirement is "only in the uae" or has to have the primary copy in the UAE
You can always selfhost
... in the UAE, so only less risky if your office is remote enough it doesn't also make a nice target (like, say, located within/near Dubai's Internet City).
2 replies →
My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical report of why they can't recover things.
11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.
> including acts of God
Serious question: how do we know something is or isn't an act of god?
https://en.wikipedia.org/wiki/Act_of_God
It's not terrorism because it's self-defence and it's not war because Donald Trump says so
It's beyond the reasonable control of Jeff Bezos. Even though he supports trump with billions of dollars, he didn't ask for this particular war.
2 replies →
I wonder if this will cause a mini cyber insurance crisis. I don't think any of those data loss plans have been tested at scale.
They guaranteed 11 9's durability, didn't they?
e: Yep
https://aws.amazon.com/s3/storage-classes/
The footnote which says that is the design durability against equipment failure literally begins:
> In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss
Even if they have payable SLA on this, most SLAs have Acts of God and Acts of War exemption.
But do they have Act of Special Operation exemptions?
1 reply →
> US refuses payout for soldiers who died in Iran 'because it isn't a war'
> In response to another question, Mr Vance rejected using the word “war” to characterise US operations in Iran, saying there was “no active shooting”.
https://news.ycombinator.com/item?id=49596955
If they haven't changed it recently, 11 nines is the durability target by design, but it is not set in any SLA. the S3 SLA is focused on availability.
It seems unreasonable to blame Amazon here. The AZ was destroyed. Are they supposed to have missile/drone defense?
I'm not going to complain to DoorDash if my order is delayed due to a car crash
If a senior leader at Doordash swore to God that your sandwich would 100% guaranteed make it to you, regardless of whether or not there was a car crash, then yeah, maybe you should complain
1 reply →
The SLA excludes force majeure.
Making a probabilistic claim while excluding a factor that dominates those statistics is... is quite creative accounting.
10 replies →
Excluding war as force majeure in the Middle East is the same as excluding high tide as force majeure building a sandcastle at low tide.
This
I don't think this has any teeth. They don't compensate in the event of loss afaict.
Considering all the data they have globally, they might still be compliant.
They say it's "designed for" 11 9s, not guaranteed.
But if they want to design for extreme probabilities you need to account for tail risks, so their design should have included a missile defense system. At some point you need to start worrying about asteroid defense too.
1 reply →
Hot take: if you have a service that is 11 nines reliable, but there is an underlying component whose reliability is lower, cap the nines to that component.
If all your backups are with AWS you don't have backups at all.
They can recover most of it? I assumed that those AZs had been offline for so long because they were like gone gone.
I suppose this is why Microsoft and Google want to build data centers in the Netherlands. Its not cheap but it is safe.
No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.
That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.
Now if this was multi-AZ S3 or whatever then this would be significant.
The article does not tell us what products were impacted.
I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.
I get that S3 is convenient and reasonably performant, but it is not cheap at all.
7 replies →
> EBS data data (sic) is unrecoverable, and that's what's reported.
I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."
> The article does not tell us what products were impacted.
… right … which conflicts with EBS being what's reported …
(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)
The more dramatic contingency you have to plan for, the more expensive the plan gets.
Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.
> to-do list becomes securing weapons, vehicles and fuel.
I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.
6 replies →
I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.
He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.
"Daddy, where were you when the flames reached our house?"
"I was in the office, reviewing Terraform plans"
Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries
A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.
1 reply →
Typically 300 miles geographically but could be hard in some Gulf States
6 replies →
But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.
If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.
Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.
Nothing is ever real-time. Eventual consistency leads to some data are not backed up.
You talking as if this is some mom-and-pop shop that you run.
That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.
Did local laws permit?
You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.
[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.
Not all data is allowed to leave all countries.
what a silly notion.
If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.
Isn't S3 claiming eleven nines of data durability?
https://aws.amazon.com/s3/storage-classes/
"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."
I wonder if "can't restore some data" includes any S3 data?
I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...
1 reply →
Well… they have a good excuse.
And witness
├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3
Ok claude, now format that in a way humans can read
Sorry, I accidentally posted it twice while fixing it and on a slow connection.
3 replies →
Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?
Did they say its S3 data? Could also be single-az EBS or RDS.
Obviously what you understand is different from the reality after you actually follow all the footnotes.
I think so.
Although the more paranoid AWS customers who turned on (and pay for) S3 cross region replication or similar cross region DR for other services would be fine.
For me-south-1 (Bahrain), all 3 data centres providing the redundancy were blown up by Iran.[1] The redundancy was localised to small geographic area and a single government--something customers of AWS were hopefully aware of when they entrusted AWS with their data.
It's always buyer beware for any claims of availability. Engineers completing a FMECA[2] will (or should) always state upfront what type of failure modes they've deliberately excluded (such as meteor strike) or else every FMECA would be full of failure modes that have never been measured, and are not worth anyone's time worrying about. These exclusions vary by application--a time capsule, seed vault, etc are intended to outlast wars and collapses of empires. Typically a bunch of data centres aren't designed to withstand such failures.
I do think however it'd be reasonable to include the prospect of war for calculating data centre / cloud service availability. Especially in a place such as Bahrain where the country is obviously concerned enough about the prospect of war to have built very permanent and expensive air/missile defence sites. New Zealand on the other hand--maybe not so important to consider.
[1] https://en.wikipedia.org/wiki/Failure_Mode,_Effects,_and_Cri...
As far as I know, the attacks happened at different times. If Amazon knew that they had lost some data redundancy, shouldn’t they have been quickly mirroring that out of the region?
5 replies →
AZs weren’t meant to be disaster resistant, eg, an earthquake or hurricane could take out a whole region.
Regions were always the scale of disaster isolation on AWS.
1 reply →
The data center layout should look something like this:
Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.
If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.
See also previous discussion: https://news.ycombinator.com/item?id=49033240
They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.
I wouldn't be surprised if the engineers said "we can probably recover between 20-30% of the data but it will cost 200 hours of engineering and the data will be 7 months old by then" and the beancounters said "we'd rather have one news cycle rather than the news watching what we can and cannot recover + save those 200 hours, we'll just say it's all gone".
I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?
So many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.
Seems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.
There are many regulations over there which prevent data from leaving the country.
Sometimes those rules can have serious consequences.
"..even if something happens only once in a billion requests, that means it happens multiple times per day within S3."
but one in a trillion...
Guess those multi-AZ promises have an asterisk when actual missiles are involved. Makes you double-check your own off-site backups.
Hey but that's exactly as per design. It is the customer's responsibility to store stuff elsewhere as DR backup, not AWS.
That's the excuse they'll say, yes, then we quote back to them "eleven nines" and they come up with an excuse for that too
What a nightmare scenario to tabletop. How do you even begin to recover from something like this?
For providers that just act as middlemen, I assume data that doesn't have a residency requirement is stored outside, so probably
a) letting customers in other areas know that their data is backed up to another continent
b) asking the AI model of your choice to translate the following into PR-speak: "Because of the boneheaded data residency requirements in this country, all your data is gone, and we weren't able to do anything about it - here's an empty copy of a re-setup version of whatever infrastructure we provide, glhf setting up everything from scratch, hope you had backups"
For customers who use such a provider or operate primarily in that area: Restore from local backups, or tell whoever depended on you that everything is gone and if you really didn't have backups, probably close up shop.
Backups in a different region?
Works unless local laws specifically block you doing that which they do for some classes of data in some countries.
Multi-cloud in the same country (if that exists in the country and is far enough apart) maybe.
5 replies →
Since roughly zero of the articles I've seen link back to the source of their quotes, here it is: <https://status.aws.amazon.com/#multipleservices-me-south-1_1...>
We collectively gave this company trillions over the years and they still can’t get it right.
us-east-1 is finally looking pretty stable for once.
Ahhh the downside of “data sovereignty” rules and the de-globalization meme strikes again. I’d bet a bajillion dollars the reason this happened is due to government thinking it’s a good idea to make it illegal to store data outside their country.
Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi-region residency…as long as the European Commission doesn’t stupidly keep trying to make encryption illegal too!
Hint: Russia absolutely knows where your data centers are.
The EU is big enough to have several geographically independent data centers. without leaving the sovereign territory.
Yes there’s never been division or wars in Europe and the EU will always exist…
Germany and France in a debt spiral and turning inward/hyper-nationalist while massively re-militarizing means the EU is a safe place to structure your data with zero redundancy.
I wouldn’t bother worrying about key industries and functions, since, as history has shown, the EU is a bulletproof institution that no country has ever left.
And of course every data center in Europe has Israel-grade air defenses, it’s not like they are just sitting ducks for a fleet of drones to take out within 24 hours.
Everything not saved will be lost.
Backup both locally and everywhere no matter what.
50%+ of companies that lose all of their data go out of business in 6 months.
DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.
And a friendly reminder that replication isn't a tested data backup.
I'm sorry but your data is in another castle.
This should have been in that super Dario game.
Uh.
Uh-oh.
It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.
The wording certainly seems very careful:
"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"
It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)
You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.
Bezos hasn't lost anything from this. He's only gotten richer.
He had little choice. The Trump tariffs could've been a massive, massive blow to Amazon, so I'm sure he felt he had to get out in front of them and buy some influence with the incoming administration.
See also Tim Cook. Doesn't make it right to suck up to Trump, but it was, and unfortunately still is, a rational move.
[dead]
[dead]
[dead]
[flagged]
[flagged]
Israel had been trying to get every president to bomb Iran for decades. There's a reason they wouldn't do it themselves. They finally got a president stupid enough to listen.
[flagged]
This is not exactly a nuanced view of the conflict, and in either case, the fact that you don't like that someone on the other side of the world is chanting death to America doesn't give you a bonus card for a free attack.
Seriously, it's like people, when deciding whether to launch a war or not, are not thinking "how will the other side react and will this conflict benefit me" but instead they are only thinking "does this nation deserve to get hit".
Well, news flash, your moral outrage does not translate into you not suffering more than your opponent during a conflict. It's a completely separate issue, and a personal issue between you and your priest or rabbi. When it comes to starting wars, you have to look at military capabilities and long term outcomes, not "does this nation deserve to be attacked".
2 replies →
> You can't have a bunch of deranged mullahs threaten the entire region and world with missiles or nukes.
Can and do. It's called the United States of America.
> This is the first administration in a long time with the cohones to do something about it.
Containment worked WAY WAY better than "doing something about it" and surrendering the strait to them. They "solved it" by aerosolizing asbestos everywhere and still have no cleanup plan. Sometimes containment works much better, which is why intelligent foreign policy worked the problem from that angle.
Wasn't victory declared 1 year ago and also a number of months ago?
2 replies →
You mean the same people who were told to keep the hostages until after Reagan got elected?
The same people who bought weapons sold by the Reagan administration to fund the Sandistas?
That’s some revisionist history you’ve got there.
Do you know why Iranians were chanting "Death To America"? Because we toppled their democratically elected government in 1953 and installed a brutal dictator so that we could continue to pillage their oil.(1) When the mullahs finally toppled that dictator in 1979 they had reason to be angry with us. This is especially true because almost immediately afterwards, in the 1980s, we used our proxy Saddam Hussein to launch a war against Iran, in which over a million Iranians were killed. During this time we provided Hussein with the means to make both chemical weapons and biological weapons (2), which Iraq used extensively against Iran. I suspect if another country did the same to us, we'd be chanting "Death to XYZ" too.
(1)https://en.wikipedia.org/wiki/1953_Iranian_coup_d%27%C3%A9ta...
(2)https://irp.fas.org/congress/2002_cr/s092002.html
2 replies →
[flagged]
Can you post the specs on your missile defense system for your home lab?
How are you dealing with the rise of low-cast swarm attacks from drones?
Is it land-based, sea-based or space-based and at what point of the trajectory do you target and do you use jamming?
If you have servers on multiple continents that's enough to become immune to attacks like this. You don't need missile defense.
I would but then I'd have to kill you.
Eh I get your point but would point out that distribution does mitigate the risk here, having all your data in DC's that can be seen from space also isn't a panacea when your next door neighbour targets them in retaliation for what your ally did.
this is reductionist to the point of absurdism.
It can be true that using cloud storage, using managed services, and paying a premium is still worth it for a lot of people and organizations, and while not perfect, still a hell of a lot better compared to the fully in your control tape backups that you distribute to different physical locations every week.
I'm not a particular fan of relying on one provider or vendor lock in, but to pretend they don't provide a service with failure rates that are low enough to be very useful is a very short-sighted take.
What's you point? That if you had run your own DC in that region (because that was your business requirement) then you'd have better missile defense than AWS?
Or maybe AWS or DIY, you are always responsible for geographic diversity?
Anyone losing data over this lost it because they'd literally told AWS to only store it in one place.
You don't need better missile defense than AWS. You don't need missile defense at all because you won't be a target. 99.999999% of the land has no missile threat on it. You are actively increasing the threat to your business by running it on the same servers that military contractors run their software on.
7 replies →
The point is that AWS has the same problem as Wildberries.
There is no difference at all between Wildberries and AWS data centers.
If you don't know what Wildberries is then go watch their facilities systematically destroyed on YouTube - centralisation is a target.
If your organization runs on AWS then you should have a contingency plan for the data center being destroyed by drones. Is that on your risk management plan?
1 reply →
I read that as a statement that says something along the (nowadays very tired) line that the cloud is not magical, it's just other people's computers.
> What's you point?
Not the OP, but the point is that decentralized infrastructure is a lot more resilient to attacks of any kind.
1 reply →
How about not bombing other countries and then acting surprised when retaliation happens? I mean clearly the problem isn't AWS as such - it is the problem that someone leading a country is totally clueless about the world. Only personal profit is in the interest of the orange clown.
1 reply →
[flagged]
1 reply →
This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.
The EU is big enough to house multiple regions for multiple cloud providers, all in the same jurisdiction (so they can actually be used within data sovereignty requirements). Not true for most of the other places in Asia/UK/South America/etc.
I'm comfortable enough with the Sydney and Melbourne AWS regions - about 700km (400 miles) apart and with (at least) 3 AZs in each. If something takes out enough AWS datacenters to lose some of work's or client data stored across all that, the uptime and resilience of the CRUD platforms I'm responsible for will not be very high on my personal priority list. (At least on AWS datacenter is within 10km of my home. I'm hoping that well before Australia gets involved in the sort of geopolitical conflict that might mean missile strikes against civilian infrastructure, I'll have headed bush to hang out with my off grid friends)
I don't think this is a flipside, unless you're thinking from the PoV of data itself rather than its owners.
data residency requirements in the EU don't categorically exclude data storage in other countries. The EDPB explicitly recognizes encrypted backups, for example, as valid as long as the keys remain in the EU and there's a secure transfer mechanism (p. 30) exactly for reasons such as disaster recovery.
https://www.edpb.europa.eu/system/files/documents/2021-06/ed...
If you can not restore data from EU based Amazon Datacenters because it’s destroyed … you will definitely have better things to do like packing your go bag or buying the last groceries for a while.
Ah yeah, the EU, a historically stable area.
1 reply →
For long-term backups you want offline cold storage in an underground facility in a friendly jurisdiction, not a datacenter.
We've beer-o-clock wargamed this a bit.
If I had an "important enough" client, I think I'd store all out local (Sydney + Melbourne AWS cross region) data to AWS Singapore (to protect against Australian jurisdictional and political risks) and to a non AWS cloud provider in the EU somewhere. I reckon thatd be close to as resilient a pile of hard drives in an underground bunker, for significantly less setup and ongoing cost, while also being much more available when needed. (Can you imagine the queue at the underground bunker when multiple AWS regions get bombed? Or even imagine getting to the bunker in "a friendly jurisdiction" while a shooting war is taking place?)
We haven't worked out a decent solution to Visa and Mastercard payment network going down for more than a couple of cloud billing cycles though.
1 reply →
War did not randomly came. America intentionally caused it.
And has lawless goverment and unaccountable tech industry making it bad place for data.
I wonder if Amazon can sue the US govt bc they basically caused this material loss to their business. I'm sure they cannot. Maybe a lawyer can explain why?
5 replies →
[flagged]
4 replies →