Iran strikes leave Amazon availability zones "hard down" in Bahrain and Dubai

4 months ago (bigtechnology.com)

This may have been long discussed, but I feel like this war is the first time I've really thought hard about how big a target data centers would be in any sort of modern peer war and how that's an entirely new thing since the last time it was really on the radar (end of CW) right? We've built trillions and trillions of dollars in infrastructure in the peace time since, and it seems fairly concentrated. AWS is amongst the biggest there is, and according to mappers like [0] there are only around 240 operational total worldwide with another 130ish under construction. Like, in one respect that seems like a bunch, but vs the kind of attacks we see done in a matter of days in modern wars it's a pretty small number for the whole planet isn't it? In the first 24 hours of the war the US and Israel launched on Iran, they hit something like 1500-2000 targets. How hardened are the data centers? Are they in structures that handle some level of explosives? Do they have counter measures like internal blast walls dividing things into cells so a few hundred pounds of high explosive in one area doesn't damage outside the cell? I mean, of course like all data centers they'll have considered extensive countermeasures to fire, environmental threats, grid issues and so on. But has "nation-state level attack via mass drones or bombardment" been part of the threat model over the last few decades? Hardening of telecoms was certainly considered for old Ma Bell and such back in the CW days but that was a very different environment.

I guess it makes me think about what a soft underbelly this could be for a lot of modern society. There's always been consideration of threats to refineries and power stations and industrial production and all those big metal deals. But like, forget any sort of nuclear exchange, any sort of crazy super Starfish style big EMP, just purely a few thousand drones nailing data centers. Nobody even directly dies, just a lot of wrecked computers. What would be the cost of losing all the clouds and colo stuff? How long to replace, at what cost? How much depends on it?

----

0: https://www.datacentermap.com/c/amazon-aws/

  • Instead of targeting data center itself, it's far easier to target the electrical substation that powers the datacenter. It's relatively simple to do. Transformers require oil to cool themselves, and if the coolant reservoir is damaged, then they overheat and shut off. This exact infrastructure attack occurred in North Carolina in 2022 [0], where someone fired bullets into the coolant reservoirs and caused a several day power outage. The perpetrator was never caught. It's speculated a foreign actor did this to gauge the response in a future wartime scenario.

    Most data centers have a dedicated electrical substation that powers it, so it's possible to target the data center without affecting anywhere else.

    [0] https://en.wikipedia.org/wiki/Moore_County_substation_attack

    • >Instead of targeting data centers, it's far easier to target the electrical substation that powers the datacenter

      That has a lot of collateral damage that may or may not be desirable though. Simultaneously it might have quite a different long term effect right? If all the actual computers are unharmed they can be powered in other ways in an emergency, even if at much higher cost. Or powered back up later, the time lost might be militarily very significant but they're not gone.

      But how many people and companies actually have full functional decentralized clones of all programs and data? How many people and companies have devices that are locked to remote hosts they expect to check in on at least once in awhile even if they're not "cloud dependent"? What if all that was literally gone, a few thousand missiles or drones and data centers are all just completely erased including tape backups, everything, suddenly we're in a world where all that compute and data is poof. And without hurting anything else, no traditional war crimes either, no power or direct food or transport disruptions. Everyone is fine and healthy, except with this huge societal exocortex gone.

      6 replies →

    • > where someone fired bullets into the coolant reservoirs and caused a several day power outage.

      So you mean to say, one doesn't even need drones, a datacenter could be (temporarily) taken out with a handgun?

      2 replies →

    • It’s far more difficult to replace a data center than to replace transformers. Ukraine’s electricity grid has been under attack for years and manages to replace and rebuild transformers and restore power within hours.

    • That wasn't thought to be due to a foreign actor though, more likely it was domestic terrorism. Why would the effect on a rural local power station ever be a good measure of a wartime scenario at all?

    • hitting the electrical grid is a war crime, not so much an AWS server farm. optics / progressive escalation does matter

    • You're barking up the wrong tree.

      The gear to replace the power infra is more readily available than the thousands and thousands of miles of wire and fiber in a datacenter, plus all the equipment, batteries, inverter/chargers, maybe some diesel generators, etc.

      If you want to do economic damage, you hit the datacenter.

      If you want to turn the people of the country against you and mobilize them, then you hit the power infra.

  • In any significant war the Internet is going to go down. That's what has happened empirically in countries undergoing significant wars or social unrest, like Russia, Iran, Yemen, Ethiopia, Syria, Myanmar, and Afghanistan. While IP packet routing itself may have been designed to survive a nuclear war, there have been many centralized systems built on top of it (DNS? Edge caching? Cloudflare? Big Tech) that are essential to the functioning of what we know of as the Internet.

    If your threat model includes war and you want to have some of the conveniences of the Internet, you should make plans for how to host local copies of data and develop local-scale communications for the people you regularly talk with. The Internet is too big of a security and propaganda risk for governments to allow it to continue to exist when they are engaged in a real existential war.

    • Some systems are anycast and available in many datacenters for bandwidth cost reasons. Netflix used to deploy content caches in ISP edges.

      But any single central point of failure might break them. Things like, is this account paid? Dunno!

      1 reply →

  • Building blast resistant is a common practices for Refinery control rooms. The same methodologies can be employed for data centers as well.

    1 blast can be expensively guarded againt. However designing anything above ground for sustained barges is practically/commercially prohibitive. Underground is only option.

    PS: Civil Engineer. Designed few of those Gas explosion resistant control rooms.

  • > the first time I've really thought hard about how big a target data centers would be in any sort of modern peer war

    Given the rapid and increasing rise of AI use in actually fighting wars, I suspect data centers won't just be a big target, they will eventually be the #1 priority target. Taking them offline won't just be of interest in terms of economic damage, it will be a direct strategic goal toward militarily winning the conflict.

    • Until it is clear that the use of AI in "actually fighting wars" doesn't put senior military people at risk of never being able to leave their own country again for fear of prosecution for war crimes, I'm not so sure that the "rapid and increasing rise" is going to actually be a thing.

      15 replies →

  • Ironically,the classical target, Washington DC, is less than 25 miles down a very simple highway to Northern Virginia's massive datacenter alley. Our national defense is ultimately predicated on heavy ordnance not being able to show up undetected in this part of the world. Hence the path preferred by attackers of burrowing into Azure signing keys or ransomware attacks on the grid. Much less hardware to transport.

    • Iran doesn't, but whether or not China and Russia have nuclear subs parked off the East Coast is anyone's guess.

  • I recently wrote a little on this https://generalresearch.com/detail-oriented/how-to-seed-a-cl...

    While we're completely at the mercy of datacenters that we can colo out racks / power / upstreams from, it's a worthy discussion for any technology company that wants some amount of digital sovereignty over their presence online and ability to provide their service independent of a hypervisor / cloud provider (or even just a centralized location).

    The best option is simply to anycast from any many distinct countries that are either neutral, or unlikely to be involved with any global or regional conflicts at any given time. You don't want them getting bombed at the same time!

  • There are ways to shield data centers if one is serious about it...

    e.g.

    https://www.reuters.com/world/middle-east/oracle-opens-first...

    • Oracle are actually subletting part of Bynet's new-ish Har Hotzvim facility which has 2,400 racks but is probably power-constrained - I believe it had 16 MW when it went live, with passive provision for doubling that. Even if it's since been upgraded, that's still only 13 kW per rack which is pretty stingy these days.

      As a very rough rule of thumb building down is about 4x more expensive than building up. So probably worth doing if you're Shin Bet (who I believe also have space in the same dc), but for the likes of Oracle it's only going to be used to serve clients with specific security requirements. Think of it as a halo project - more of a marketing exercise than something that's actually going to be used by the average customer.

      The same goes for datacentres hosted in cold war bunkers etc - they always end up being too constrained in one way or another to be useful. The big facilities end up being built above ground and rely on geographic redundancy rather than trying to make themselves (literally) bomb-proof.

  • The way everything is so overleveraged on the success of these companies being packed into ETFs, it would probably take down the whole economy. You'd be able to shut down even more manufacturing without even destroying it just from economic forces. That is unless the US responds by nationalizing everything, which they won't. They'd rather it go to smithereens so someone has a chance to be made wildly rich rebuilding.

  • > I've really thought hard about

    Yeah. Financialize the economy presupposing a global open market, then subvert, boycott and bomb said market. So clever.

  • Tangentially related but Iran wasn't much of a threat to the USA before Trump decided to attack it. And apart from Israel, nobody is backing this war. The sooner he realizes it makes zero sense, the better for the whole world. It seems that apart from Russia and the USA, other countries are not so eager to start wars. And what is happening now is a bitter lesson also for China: starting a war is easy, winning it is nearly impossible. So I hope we won't really start to build all infra in under-earth bunkers after all.

  • > Nobody even directly dies

    People almost directly start dying if data centers go down.

    Not in the minutes, but within days definitely

  • Then ask yourself why is the US so aggressively trying to switch the world to a martial stance.

    It’s a rhetorical question, of course, because we all know it’s because China is winning the traditional economic game on the manufacturing the McKinsey and Bain class sold out for decades and therefore military will have to become the new leadership measuremen, only appreciating as an asset in a less safe dog eat dog world.

    The Thiels and friends who came up with this shit of course have their own infrastructure in their end time bunkers, but however stupid this gambit sounds, it’s what’s being played right now.

  • Agreed that Govt/Military runs on AWS/Azure/whatever. They care about "security" in a "virtual" sense, but I presume soon we'll see requirements like: "Must Have: Missile Defence Perimeter" next to the "Must be FIPS compliant".

    • My partner works in that space.

      Sovereignty and self-sufficiency are big topics. The US centric cloud at least is killing itself through geopolitical risks for gov customers outside the US. Literally number one operational risk now.

      1 reply →

  • hmm maybe aws should make datacenter locations secret?

    I mean, why even publish those locations?

    if this is purely for PR, they can publish fake locations...

    if this is for VIP visits... well you can always send private invitations

    • But aren't they pretty hard to hide? I mean, they cover a lot of grounds, they have lots of infrastructure leading right to them...even if someone makes a few wrong guesses, it's going to be easy to find where the data centers are.

    • >Disclaimer: Please be aware that Amazon Web Services does not list its data center locations publicly. Hence all AWS listings in our database are based on publicly available information from third parties, open databases, property registries, construction applications, permits, tenders, news coverage and our custom research. There may be incorrect or outdated locations, as well as locations missing.

      AWS doesn't disclose their locations.

    • > hmm maybe aws should make datacenter locations secret?

      Open source intelligence today rivals the best of any nation. That would be virtually impossible to hide for long.

  • >We've built trillions and trillions of dollars in infrastructure in the peace time since, and it seems fairly concentrated.

    and thus is easily defended. It would be a pocket change - tens of millions - for AMZN to put say a Rheinmetall Skyshield https://en.wikipedia.org/wiki/Skyshield at the data center.

    • Considering how hard US military bases and radar systems have been hit (and those are not city-sized target) I am unconvinced that even AMZN's pocket change could realiably protect against the kind of attacks we see in this war

      1 reply →

>Amazon tells its employees to deprioritize these regions as the Iran war deals meaningful damage to its infrastructure in the Gulf.

Deprioritised means migrate usage out of this zone just in case anyone misreads the context here.

This is the part I don’t understand about Elon’s Terafab: What protects it from a missile? Or laser?

Better yet: Jeff or Sir Richard hook up one of their ships and just tow away the Terafab… yoink!

There are good physics-based reasons to put data centers in space, but the geo-political world isn’t informed merely by physics.

  • Trying to build things in a way that they have to be resistant to missile and drone stand is not a very reasonable scaling constraint. That's why we need non stupid leaders who don't start pointless wars

The big security mantra from aws has always been that they deal with the security of the cloud and you/we deal with the security in the cloud.

I wonder if this will translate to amazon implementing para-militar security of the cloud (eg: drones to defend from drone attacks).

My intuition suggests me that:

- Bezos would have absolutely considered this, like seriously considered - the current ceo likely won’t

Btw the writing has been on the proverbial wall for some time, amazon is in their day-2 era.

Trump really only babbles nothing burgers now. The whole "we must open the Strait of Hormuz", but it was closed following the invasion of Iran at the behest of Netanyahu proxy-controlling Trump - so how is that then logical that you refer to a prior state that already existed, as a new war-meta-goal? This is like an autogenerate of fake news and lies. This can not be the person really "leading" the USA, so who is really making those decisions? Trump even forgets what he said the day before and even contradicts himself in the very same sentence; then he chains buzzwords that make no sense, such as "we can not have healthcare because we must wage war instead". This is like George Orwell 1984, but stupid. George Orwell's book made sense; Trump is just dementia 2.0 1984 reversed. Nobody would read that Trump-novel, just as nobody serious would watch Melania. It's the ultimate Soap TV show for the US audience, but it is just not watchable. No risk management or analysis; Hegseth recently mass-fired those who said his plan is stupid. Well, even after firing people, the plan is just stupid.

  • Most of the world is past blaming Trump for this. They're now blaming Americans for not removing an obviously insane tyrant from power. Americans have long said, "The price of freedom is eternal vigilance." Well, where's that vigilance? You're not all blind, are you? Suddenly lost your taste for "freedom", have you?

    You may not have voted for Trump, but now you're accountable to the world for him.

I hate when "the cloud" which I imagined to be some entity in ether space, turns out to be just a building with computers that can be bombed

[flagged]

  • It's not quite as bad as in the UAE where only about 10% of the nation has citizenship but it's basically the same picture. Most people there are practically indentured servants

if you dont colo your own servers you don't own anything.

  • >if you dont colo your own servers you don't own anything.

    I'm confused, what does ownership have to do with this particular failure mode? The issue here is a (for many) unforeseen new tradeoff involved in centralization. Colocating at a central place has the exact same tradeoff in this case: bandwidth is vastly more available and cheaper towards the core, and there are significant amortization gains to be had with a lot of basic shared infra. But it's also one big structure holding a lot of computers and infra everyone is depending on, that's the whole point of it! We're all sharing network backbone and power filtering/redundancy and so on and so forth, vs paying for that separately. That means a missile or drone or bomb hit to the building still hits all of us whether we own the servers there or we're running workloads on someone else's servers.

    The only responses are either central counter measures or decentralization. Both have significant costs and complexity, that's why it wasn't just done proactively right?

  • You should have the opposite takeaway - if you don't have redundancy in the cloud you don't actually have uptime.

  • I don't think co-locating with AWS or any other DC in Middle-East would help in this case. Unless you bring your own missile defence network, you are vulnerable.

    In the case of if you could bring your own missile-defence-network, then you probably don't need co-location anyway. (There is nothing "co", it's just location you build & operate, with your Patriot or whatever)

    • boolean "you are vulnerable" means nothing, because it's always True

      spreading out decreases risk, concentration increases it