← Back to context

Comment by thanatos519

8 days ago

Backups should be far away, too. Apparently some companies lost everything on 9/11 because their backups were in the other tower.

Some foolishly believed that the twin towers were invincible after the 1993 WTC bombing.

Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.

  • Jersey City still was fine and 50 miles can be problematic for certain types of backup (failover) protocols. Regular tape backups would be fine but secondary databases can't be that far away (at least not at the time). I remember my boss at WFC saying that the most traffic over the data lines was in the middle of the night due to backups - not when everybody was in the office.

    • Companies big enough will lay the fibre. 50-100 miles of fibre isn't much if you are a billion dollar business. Even companies like BlackRock who had their own datacenters have since taken up Azure. 50 miles latency is negligible, even for databases.

      19 replies →

  • Before 9/11, most DR (disaster recovery) sites were in Jersey City, NJ just across the river from their main offices in WFC or WTC, or roughly 3-5 miles away. After 9/11, the financial industry adopted a 50+ miles rule.

    IIRC, multiple IBM mainframes can be setup so they run and are administered as a single system for DR, but there are distance limits.

    • A Geographically-Dispersed Parallel Sysplex for z\OS mainframes, which IBM has been selling since the '90s, can have redundancy out to about 120 miles.

      At a former employer, we used a datacenter in East Brunswick NJ that had mainframes in sysplex with partners in lower manhattan.

      2 replies →

  • >Some foolishly believed that the twin towers were invincible after the 1993 WTC bombing.

    I was told right after the bombing, by someone with a large engineering firm (Schlumberger or Bechtel), that the bombers could have brought the building down had they done it right.

Funnily enough, Germany has laws for where you are allowed to store backups exactly due to these kinda issues. Fire, flood, earthquake, tornadoes, whatever you name, backups need to be stored with appropriate security in mind.

  • Germany, of course. Like my company needs government permission to store backups.

    • More like: your company (or government agency) is critical infrastructure or of a certain size, so there are obligations on how you maintain your records. It’s not like the US or other countries don’t have similar requirements.

      32 replies →

    • (Without knowing the precise nature of these laws) I would expect that they don't forbid you to store backups elsewhere. It's just that they mandate that certain types of data be backed up in sufficiently secure and independent locations. If you want to have an additional backup (or backups of data not covered by the law) in a more convenient location, you still can.

      5 replies →

    • Certain data records need to be legally retained for certain amounts of time; Other sensitive data (e.g. PII) have security requirements.

      Why wouldn't government mandate storage requirements given the above?

    • No it doesn’t. It does however need to follow the appropiate standards commensurate with your size and criticality. Feel free to exceed them.

They deserved to lose everything... except the human lives, of course.

That's like storing lifeboats in the bilge section of the ship, so they won't get damaged by storms.