Comment by adrian_b
4 days ago
There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a greater amount of data than the added redundancy.
For instance, on Linux there is par2cmdline. For all my backups, I create pax archives, which are then compressed, then encrypted, then expanded with par2create, then aggregated again in a single pax file (the legacy tar file formats are not good for faithfully storing all metadata of modern file systems and each kind of tar program may have proprietary non-portable extensions to handle this, therefore I use only the pax file format).
Besides that, important data should be replicated and stored on 2 or even 3 SSDs/HDDs/tapes, which should preferably be stored themselves in different locations.
Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.
Huh?
The issue being discussed is random blocks, yes?
If your entire drive is bricked, that is an entirely different issue.
Here’s the thing. That SSD controller is the interface between you and those blocks.
If it decides, by some arbitrary measurement, as defined by some logic within its black box firmware, that it should stop returning all blocks, then it will do so, and you have almost no recourse.
This is a very common failure mode of SSDs. As a consequence of some failed blocks (likely exceeding a number of failed blocks, or perhaps the controller’s own storage failed), drives will commonly brick themselves.
Perhaps you haven’t seen it happen, or your SSD doesn’t do this, or perhaps certain models or firmwares don’t, but some certainly do, both from my own experience, and countless accounts I’ve read elsewhere, so this is more common than you might realise.
20 replies →
Thank you for this.
I had no knowledge of pax, or that par was an open standard, and I care about what they help with. Going to switch over to using both in my backups.
For handling pax archives, I recommend the "libarchive" package, which is available in many Linux distributions, even if it originally comes from FreeBSD.
Among other utilities, it installs the "bsdtar" program, which you can use in your scripts like this:
And for extraction:
The bsdtar program has options for compressing and/or encrypting the archives, for the case when you do not want to use directly other external programs.
"par2create" creates multiple files from the (normally compressed and encrypted) archive file, for storing the added redundancy. I make a directory where I move those files, then I use a second time bsdtar (obviously without any compression or encryption) to aggregate those files in a single archive with redundancy.
The libarchive package can also be taken directly from:
https://github.com/libarchive/libarchive
"libarchive" handles correctly all kinds of file metadata, e.g. extended file attributes and high-resolution file timestamps, which not all archiving utilities do. Many Linux utilities, with the default command-line options or when they have not been compiled from their source with adequate compilation options, which happens in some Linux distributions, may silently lose some of the file metadata, when copying, moving or archiving.
there's no reason that you have to create multiple files for par2 if you are storing the recovery data with the protected data. It only was split into files of varying size due to its source in protecting usenet posted binaries to allow users to not have to download the entire recovery data when they only needed a portion.
This is fine, but I'd prefer an option to transparently add parity bits to the drive, even if it means losing access to capacity.
Personally, I keep backups of critical data on a platter disk NAS, so I'm not concerned about losing critical data off of an SSD. However, I did recently have to reinstall Windows on a computer because of a randomly corrupted system file. Which is something this feature would have prevented.