Comment by TacticalCoder

When I took charge of solving backups for the single important box with unique, irreplaceable data -- the accounting system -- at an SME a long time ago, I think I approached it with the right amount of correctness. Therein, losing a day or three of recent data would have been recoverable; losing all of it would have been catastrophic.

I devised a system to perform bare-metal backups onto an easily-swapped, external 2.5" hard drive, using Acronis. I provided a plurality of these hard drives, and they were to be rotated off-site. The system was tolerant of human error and would proceed with making valid, current backups even if the drives were rotated incorrectly, or if not rotated at all on any given day. The backup drives each had complete file history (yay shadow copies) from an ever-advancing date, so any given drive could be used as a time machine of varying resolution, and also as the single source from which to independently start fresh.

I'd watch the logs to see that it was done, and for the most part: Whoever was assigned to that role normally did it properly-enough.

I documented it and showed the other technical folks how it works.

Sometimes I'd wander back and make sure the backup drives weren't accumulating on-site (there should never be more than 2 on-site). I'd periodically test these backups by restoring them completely onto identical hardware, to make sure the system hadn't got crufted up somehow and that it still continued to perform its task of restoring a working system from zero.

It worked fine for years and years. We never had to use that backup, but I had every confidence that it would be useful if that ever became necessary.

Eventually, my role changed and those things rather officially became Not My Problem.

Later, they moved the accounting system from that lineage of stout Proliant boxes to a trash-tier small-form 1u Lenovo machine that someone found used, on eBay, for cheap.

Backups are handled by the clown, somehow. The last I heard anything about it, the person doing the talking was very pleased with the money they'd saved and that they'd no longer have to pay "extortion" to Acronis.

I have every expectation that nobody has ever restored these backups. They're probably relying on the sheer hope that they'll never have to restore them, much less from zero.

And I also hope they never have to restore them, lest they may find out exactly what that data is worth to them.

> How does a medium-sized SME were all the payrolls depends on Sara and her USB stick do if, literally, their servers do catch fire.

Like every job, we overestimate our importance.

What do they do? They pay everyone the same as last month as a temporary measure, ask you to talk to your manager if your pay should be more this month, warn everyone that they're going to recalculate the payroll and adjust any differences next month. Then they calculate everyone's pay from the inputs, which really isn't such a hard problem when the alternative is failure. Maybe they pay some fancy consultants or an SAAS provider for a few months. Maybe they have to cut a few corners. Maybe they even get fined by their state's DoL. Life goes on.

> How many SMEs out there are depending on Sara's knowledge of the USB memory stick and how to use it?

I think at least in part, that is the point: orgs are missing the part of the equation where the institutional and organizational knowledge is critical. Sure, the code to accomplish parts B and C can be re-duct-taped together in a month or so by off-shore, or maybe an agent... but part A, its plumbing, and why it does what it does the way it does it due to historical failures and the knowledge behind that is probably what keeps it going.

Those things are learned starting at the ground level by bumping into them in the trenches.

The company just shuts down and its customers switch to competitors. This is economically efficient. The redundancy of a company is another company. It's a bit like how we don't insist on every server running two CPUs in lockstep in case one fails, because we have more than one server to handle requests.

>The USB stick hints at a big problem in our trade though: how do you "reboot" your IT infrastructure if it literally burns to the ground? I'm not talking about Google-scale systems (which still couldn't restart from scratch IIUC but they're actually working on it?) but only about SMEs.

Maersk ground to a halt because it got done nearly 100% by cryptolocker. IIRC they went to hard copy records, called everyone, got all of IT together with some company credit cards to get new laptops and flash drives and shit and literally rebuilt their infra from scratch.

https://www.itnews.com.au/news/maersk-had-to-reinstall-all-i...

I read a better post mortem but thats the highlights.

>How many SMEs out there are depending on Sara's knowledge of the USB memory stick and how to use it?

Part of my day job is finding, documenting and remediating these sort of issues.

"The CEO Coded this application in VB5 15 years ago, the entire business relies on it, theres no source code, theres no binary backups and the one computer it runs on just had its PSU fail"

"Theres a cron somewhere that compresses, zips and transports the payroll database interstate, outside of our network, before our weekly pay run"

"Theres been no documentation of this environment for 20 years, most of the hardware is that old, and the team that developed it just sold all their shares and left"

This shit is my life lmao.

Theres obviously some bias, because the good companies aren't asking me to do it for them. But I make a decent living examining, documenting and remediating this shit.

> How does a medium-sized SME were all the payrolls depends on Sara and her USB stick do if, literally, their servers do catch fire.

The SpecOps guys have the following bit of wisdom on offer: "Two is one and one is none".