Comment by bane

9 days ago

Goodness, I have over 100TB at home and it cost less than a two or three thousand dollars to put in place. That's like $25 per TB.

> The stored data amounts to 858TB (terabytes), equivalent to 449.5 billion A4 sheets.

No, the 858TB amounts to under $25k for the government of the 10th largest economy, of one of the most sophisticated countries on the planet, to put in place.

Two of those would be less than the price of a new Hyundai Grandeur car.

> “It’s daunting as eight years’ worth of work materials have completely disappeared.”

So they're clocking in at around 100TB/year or 280GB a day. It's respectable, but not crazy. It's about 12GB/hr, doable with professional, server level hardware with backup moved over dedicated fiber to an offsite location. Multiply the price 10x and you can SSD the entire thing.

Even with data sovereignty consideration demanding an entirely 100% home grown solution rather than turning to AWS or Azure, there's no excuse. But it's not like the cloud providers don't already have CSAP certification and local, in country, sovereign clouds [1] with multiple geographic locations in country [2]

South Korea is full of granite mountains, maybe its time the government converts one into an offsite, redundant backup vault?

1 - https://erp.today/south-korea-microsoft-azure-first-hypersca...

2 - https://learn.microsoft.com/en-us/azure/reliability/regions-...

25 comments

bane

Reply
Nifty3929  9 days ago

~1PB of data, with ingestion at a rate of 12GB per hour, is a tiny amount of data to manage and backup properly for a developed world government. This is silly. Volume clearly should not have been a hinderance.

Backup operations are often complex and difficult - but then again it's been worked on for decades and rigorous protocols exist which can and should be adopted.

"However, due to the system’s large-capacity, low-performance storage structure, no external backups were maintained" ... "the G-Drive’s structure did not allow for external backups."

Clearly [in]competence was the single factor here.

This is what happens when you come up with all kind of reasons to do something yourself, which you are not qualified to do, rather than simply paying a vendor to do it for you.

  • lloeki  9 days ago

    > Backup operations are often complex and difficult

    It quickly becomes much less so if you satisfy yourself with very crude methods.

    Sure that would be an imperfect backup in many ways but any imperfect backup is always infinitely better than no backup at all.

jart  9 days ago

The most sophisticated countries and companies are smart enough to use the least sophisticated backup methods. SK needs to backup their data to cassette tapes and tape libraries cost a bit more than that, but not much. Even if they boat their tapes over to an iron mountain in the US, I can't imagine the equipment and service fees are going to cost them more than a few hundred grand. They'll be spending more on the headcount to manage the thing.

  • hedora  9 days ago

    The operational expenses of this stuff dwarfs the hardware cost. For the tape mountain, you need robots to confirm the tapes still work (mean time to detection of device failure and recovery are key for RAID durability computations). So, someone needs to constantly repair the robots or whatever.

    If I was being paid to manage that data set, I’d probably find two enterprise storage vendors, and stick two copies of the data set on them, each with primary secondary backup. Enterprise flash has been under a dollar a gigabyte for over a decade, so that’s under $1.7M per copy, amortized over five years. That’s $700K per year, and one of the four copies (at 3-4 sites) could be the primary store.

    (I can’t be bothered to look up current prices, but moore’s law says there have been six capacity doublings since then, and it still applies to flash and networking, so divide my estimate by 2^6 — so, ten-ish grand per year, with zero full time babysitters required).

    • chii  9 days ago

      even with dual vendors, you'd have to still put in place a backup/restore procedures (with the associated software, which may need to be custom). Then you'd need regular testing. These operational concerns will basically double the cost yearly, probably.

      1 reply →

Den_VR  9 days ago

But it would not have been $25k, it would have been 1-2 million for an “enterprise grade” storage solution from Dell or a competitor. Which isn’t much compared with your granite mountain proposal, nor with the wages of 750,000 civil servants, but it’s a lot more than $25k.

samus  9 days ago

The article reads like they actually have a fault-tolerant system to store their data. This is probably a data dump for whatever files they are working with that might have started out as a cobbled-together prototype that just picked up momentum and pushed beyond its limitations. Many such cases not only in government IT...

  • bane  9 days ago

    Looking at the article, my read (which could be wrong) is that the backup was in the same room as the original.

    • lazyasciiart  9 days ago

      No. It says that "most systems" in this data center are backed up to separate hardware on a different floor, and then a backup is made at a physically remote location. This particular G-Drive system was not on the standard backup process - it sounds like it was much higher volume than any others, so maybe they couldn't use it. They did have a pilot going to get G-Drive backed up...it was supposed to be scaled up to the whole thing in December.

      3 replies →

WalterBright  9 days ago

You can buy a 24Tb drive on sale for $240 or so.

Sometimes I wonder why I still try and save disk space :-/