Comment by ocd
9 hours ago
I have external parity for everything with rsync. I don't have any storage that isn't matched with external backups. Fortunately I've never seen any bitrot or had a drive fail on me since I tend to replace them before they do. And yeah, I could add in some extra HDDs for RAID, but this is more a personal choice because of how much I hate SATA.
Filesystems like ext4 are because I value boring and rock solid stability over semi-experimental status of modern filesystems with all the features.
What do you use for external parity? Par2?
If you were to replace your current drive with a new larger one and your RAM or your SATA controller silently corrupted the files how would things play out? Would these corrupted files propegate to the backups?
Ext4 is battle tested so I understand your reasoning. I think you just need to figure out a way to detect silent corruption and a way to snapshot your files in case they do get corrupted.
From experience I can tell you that it is an absolute pain having to manually sort through a bunch of files trying to detect which ones are corrupt and which ones are good.
Misspoke. I didn't mean that type of parity, but just hashes, and a matching drive (in make and capacity) externally for an exact copy.
How do you know you haven't seen bitrot? What rsync arguments do you use to error out when an old, untouched file suddenly changes?
I suppose I couldn't guarantee I haven't, but I keep a total backup, and other backups based on file lists. I keep a record of every file of importance that I've ever written/modified/interacted with, and those lists themselves are also git managed.
I compulsively look at the rsync backup running to see what's going to transfer, and typically do a dry-run first. That's no perfect solution, but it works for me. If I see something odd I don't quite remember, I check the hashes between both drives, and every time it turns out it was just a path change since I try to keep my data as organized as possible.
rsync arguments are just the plain `--archive --acls --xattrs --verbose` and depending on which backup I'm doing, `--recursive`, `--delete`, `--files-from=`. Nothing other than vanilla at all.
What do you dislike about SATA?
The cables and how they have to be managed/positioned in a case. I've had panics over a drive failing when it was just the cable after I'd already replaced the drive, restored data, and ran tests on the old drive that turned out to be fine each time.
rsync does not protect you from bitrot unless you have some script that specifically collects the rsync log and tracks when existing files change, and you somehow know every file that should not have changed and get alerted that a file changed when it shouldn't have.
Even with that, that is still just 2 equally authoritative copies, without any way to know which one suffered the bitrot.
Ok the tie-breaker data could come from history. Yesterdays log and the day before gives you the extra data points to say which side changed. As long as the log itself is somehow above reproach.
Except this script is a myth anyway. You would still need to have something where you tag files as "this file shall never change again, so if it ever appears to, tell me so I can ok or reject the change." and you would have to actually do that tagging and reviewing.