The novice said: "I will save my working files, but not my system and application files, as they can be always be reinstalled from their distribution disks."
The master made no reply.
The next day, the novice's disk crashed. Three days later, the novice was still reinstalling software.
If that's your situation, it might be worth backing up applications. However, there are better solutions that make this irrelevant. For instance, using NixOS, you can literally back up a single text file describing your system setup, and you can recreate the system from it automatically in minutes, guaranteed to be bit for bit identical.
In my experience that works in box like 80% of the time. Which is great! But a backup is great in case it doesn’t.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
At the same time it is significantly more "hardware-change" resilient, since the same backed up config file (minus perhaps some very system specific lines that you may or may not have) can bring up a functionally equivalent working system on quite distinct other hardware as well.
If your NixOS setup is not of hello-world kind, you typically have sops in it (or another secret management tool) and want to backup your age keys as well. They're necessarily separated from the main config tree so they don't end up leaking to the globally accessible nix store, and in the most typical case they're located in a 600 file outside the git repo. Forgetting to back it up is a classic NixOS gotcha and also a generalized version of this koan.
Wouldn't you just re-encrypt the secrets with a new host key? That's what I usually do (boot once to generate keys, update the secrets, rebuild and switch remotely to the machine)
If bare metal recovery takes shorter time than reinstalling, then you should make bare metal backups. When bare metal updates, so should your backups.
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
You've missed the point. It doesn't matter that you can rebuild the system exactly, the example in question had little to do with how accurate the restores were, it had to do with the time to reinstall.
I too can entirely rebuild many systems simple from kickstarting a new one and applying an ansible playbook. Do I want to? No, that's too slow, and I might be doing this operation for tens or hundreds of systems.
We have multiple levels of backups. We use veeam for VM and physical backups of the entire system to a disk store that's also shipped off-site to an immutable store, but we also run an older system based on rsync which rsyncs the system to directory on a server (sans some data directories) that uses snapshots to keep space usage minimal for the purpose.
In many cases, if we want to restore we can literally just rsync data back to the host, whether one or a few files or most of the system.
I can restore many types of failures within seconds or single digit minutes at most, and the time required is mostly unattended time. Time to restore is often an important and overlooked metric, until you've been bitten by it and realized that an hour or more of work from a person to restore something doesn't scale well in some scenarios, and those are often the most important ones.
But it's not actually slower. When NixOS loads you get a list of all previous states to chose from. Booting to any one of them is just as fast as booting to the most recent one.
Restoring some backup via copying bits around is much slower because you have to wait for those bits to copy, whereas booting to previous configs is just rejiggering references to bits which are already in place and need no manipulation.
Immutability buys you quite a lot here, although some up-front discipline is necessary to take advantage of it, which might be a dealbreaker for some.
If you think this is historical, you have a lot to learn about production servers in the wild! =)
Just as a not-insignificant portion of the world still relies on a handful of ancient AS/400 systems, a pretty big portion of society depends on the good running of hand-configured Windows Server.
Like, you can do the same with any distro and say Puppet or Ansible but only to within that distro packages, any 3rd party stuff (and people use more and more stuff that's 3rd party to the distro) might just have dead link for the download.
It does, actually! Because only the package definitions are needed, and you can get those from the tarball of the relevant nixpkgs repo commit. And since building+installing is the same Nix command as downloading+installing, whether the binary caches still have the old package (or are even still up) is only the difference between a fast download and a slower build. This is different from most package managers, where trying to install anything old means spinning up a complex build environment and learning a bunch of new tools.
Of course, if even the original sources are no longer available, you'd have to change the broken URLs. And if the source has somehow been scrubbed from the Internet you're SOL
> The master said, "Do not despair, for yesterday I took one of your backup tapes and posted it to my brother in China. He will return it."
The master should not have done this, even by sea mail, for the very first law of backup is: you will only learn to make proper backups after you suffer a catastrophic loss of data.
Protecting the student from this is depriving him of the only true knowledge, the one that is not merely stored in the head, but burned into the skin.
He loaded the week-old backup, but it was the same. Eventually, he realized that the virus had struck four months before. He returned to the master and said: "Master, you have the only uncorrupted copy!"
But the master had already overwritten it with a copy of DOOM.
> Store your Really Important Info in a Google Drive/iCloud synced VeraCrypt volume
There is no way I would entrust Google or Apple with keeping important data. Google being infamous for account issues with absolutely no support and Apple already borked my account in a way that makes my iPad unusable the moment I turn on backup sync.
That's about the right size for all the data that actually matters at a big corporation. Everything else is just marketing slop that people think is important.
That is just nonsense. Is your work experience perhaps only in marketing-oriented shops for you to think that?
Most models from architect offices of any size will be larger; many experimental results from applied physics research labs will be larger; any single project from a film production company of any size will be larger; any combined weekly security log at MSPs of any size will be larger; any industrial plant's daily production indicators will be larger; any energy company's hourly operational measurements will be larger.
Storage is Hetzner Storage Box which so far feels great. Cheap and supports SSH SFTP WebDAV, so you can use rsync or restic and browse artifacts over www.
It feels a little slow hosting in Germany but using from US. But I haven’t found a provider with the same cost / features.
NetDynamics shares some promo codes every so often, they have a data center in Utah as well as the EU. I wouldn't call it super low latency -- but backup generally doesn't need to be. I've used both at various times for a personal off site backup target.
But for commercial use, IDK, you'd want to dig into the details of their system. Not exactly S3 levels of resilience, but perhaps enough for some people, I leave it to others to judge.
I worked for a small manufacturing company some years back, and depending on what you're doing it's easy to rack up some huge file storage usage and big databases without being in the business of video, photography or music. The database that only ran our manufacturing, not inclusive of HR, accounting, etc., was about 1TB. The design files that went to the plant (most of which were just text, you could open them with Notepad) were another 3 or 4 TB worth. Maybe another 1TB or so for various collateral bits of data related to jobs, mostly PDFs.
The company has, I'm sure, probably tripled all that by now. In their case it compressed really well, but still.
The $/TiB storage cost currently creates weird choices in backup media... even making some older media formats economical again. Just finished a consumer storage option survey yesterday.
Even if you account for consumer drive+spare cost, a 8TB HDD is still the best JBOD option. For under several dozen 25GB projects, the BD-R disc drive is currently the cheapest media backup option (about 35% cheaper than HDD above 12TiB). Note 50G and 100G BD EOL media are actually equal to or a worse deal than HDD.
Also, HPE LTO9 18GB Tapes are currently around 5 to 15 times cheaper than HDD and SSD in terms of data storage cost, but the fixed cost of tape auto-loaders and drives don't always make sense until you have a significant amount of backup data. Tapes may also create a single point of failure in use... so incremental backups are a non-zero risk, and triple redundant hardware is required (old hardware gets hard to find.)
The classic joke is still true: "Fast, cheap, or good.. choose any two..."
I looked in to the bluray thing and concluded it just doesn't make sense at any level. You have to buy a bluray writer which will become increasingly rare, and then you are limited to 25GB disks where you have to factor in a chunk of that will go unused as each project won't fit on exactly 25gb.
For the effort you are expending, just getting a hard drive makes way more sense even if the $/gb is technically a little worse. The utility of hard drives is much better.
A novice asked the master: “What must I do to learn the Zen of Backup?” The master whacked the novice in the head, grabbed the keyboard, and ran sudo rm -rf /. The novice smiled and left feeling enlightened.
After six years on my old build, last week I built a new computer. One of the things I invested in was treating the backup regime as the most important aspect of the new build. There's a piece of mind one gets from achieving the "seven heads"
Agreed. I wish more ads were like that. Note that the advice is almost three decades old, and still absolutely relevant. The classics are always a hit...
I followed the link, and see that Veracity is still around, almost 30 years later, but in a very different form from what it must have been, back then.
A weird reaction. You're missing some very good advice, advice that is completely oblivious to the backup product... a now-defunct product which is only advertised by a now-defunct company, just the once (that's refreshing), very clearly and honestly (that's refreshing), at the very end of the slideshow.
I accidentally let qwen wipe my 60tb nas the other day, and the tao of that is "well at least i dont have to stress about what im doing with my 30 years of rando .docs, facebook pics, games, ebooks and star trek episodes anymore"
c'est la vie, it was only 2tb filled and mostly games
Like giving a 6yo unix jr eng a crayon and root login. "you made them ALL apfs at the same time??" sorry dad. i can't believe people use this in prod, but i know i'm also not flagging perms my pi is ready to roll! gotta watch qwen3.6, they will be getting demoted.
I spent lots of time thinking about backups. I've got files dating back to 1991 and even some from the 80s (but I don't care about those so didn't bother backing them up: when the last 5"1/4 shall stop reading, those will be gone and it's fine).
I've even got stuff like copies of early websites made by friends, websites long gone and I'm pretty sure I'm the only one to still have backups of those.
To me verifying the backup should be part of the backup procedure itself and so I did just that: my backup procedure does verify that the backups can be decrypted/unarchived and it then verifies the files inside the backup.
Now the thing is: you don't need to verify 100% of your backups all the time (as in I don't decrypt and verify the checksums of 100% of my backups all the time). You can do random sampling: over n backups, you can be reasonably sure at least x% of the files are correct.
So random sampling is part of my backup procedure.
As those are encrypted backups, the verification also ensures that the backups can be decrypted (would be too bad otherwise, wouldn't it?).
I also believe the backuping procedure should not be able to go wild and destroy data. So my backup procedure is done by a podman container that access the volume with all my data (the one that needs to be backed up) read-only.
> To believe in one's backups is one thing. To have to use them is another."
Yes, which is why a proper backup procedure verifies the backup it just created. The greenlight is only given to the backup that's just been made once it's been verified by my verification script.
Some stuff can be verified automatically: for example say you backup a Git repo: a strict, full, git fsck on the backup (that is: on the Git repo once pulled out of the backup, before greenlighting it) works fine.
For other things I've got my own verifications.
> These hundreds of corrupted files have been flowing through my backup system. Now I do not know which files are clean and which are not.
Which is why many of my files, which I know aren't supposed to ever change anymore, have a partial checksum added as part of the filename.
I don't have:
DSC0983747.JPG
but:
DSC0983747-b3-7e228491a0.JPG
where 7e228491a0 are the first 40 bits of the Blake3 checksum of the file.
Then it become very complicated for "something" (bit rot or malicious) to silently corrupt my stuff for the backuping procedure (which only has read-only access to data, remember) goes crazy bonkers and warns me as soon as two identically named files (one on the last backup and the current version) have an identical name but different content and one doesn't match the checksum (this is cause for a "stop the world" and immediate enquiry: and, yup, it already allowed me to catch a corrupted file).
I've moreover got a sheet of paper, laminated, that explains how to access the backups and that explanation is saved in many places (safe at the bank, safe at my brother's house in another country, etc.): should say my place burn with me inside, but my wife be safe, she'd be able to access the backups too.
As for my main data, it's RAID on ZFS on a machine with ECC and that's where the backups are made (and automated) from.
Proper backup procedure is not hard: it just takes some time to set up properly, once. Then it's done for a lifetime.
If that's your situation, it might be worth backing up applications. However, there are better solutions that make this irrelevant. For instance, using NixOS, you can literally back up a single text file describing your system setup, and you can recreate the system from it automatically in minutes, guaranteed to be bit for bit identical.
In my experience that works in box like 80% of the time. Which is great! But a backup is great in case it doesn’t.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
Backup the distribution files via mirror/proxy-mirror?
3 replies →
My strategy for Gentoo, more or less:
/etc/ - all modified configuration files, including
/etc/portage/ - make.conf, package configuration (package.accept_keywords, package.use, and friends)
/var/lib/portage/ - world, and world sets. basically everything portage needs to rebuild the system
/var/db/repos/ - notably local and/or self-owned package repos
/home/<username>/ - no explanation needed
/root/ - no explanation needed
This should contain enough to pretty much rebuild a Gentoo system from a clean downloadable Stage onwards without (too much) intervention.
It's not guaranteed to be bit identical (the why of it is a can of worms), but it's identical enough for all practical purposes.
At the same time it is significantly more "hardware-change" resilient, since the same backed up config file (minus perhaps some very system specific lines that you may or may not have) can bring up a functionally equivalent working system on quite distinct other hardware as well.
If your NixOS setup is not of hello-world kind, you typically have sops in it (or another secret management tool) and want to backup your age keys as well. They're necessarily separated from the main config tree so they don't end up leaking to the globally accessible nix store, and in the most typical case they're located in a 600 file outside the git repo. Forgetting to back it up is a classic NixOS gotcha and also a generalized version of this koan.
Wouldn't you just re-encrypt the secrets with a new host key? That's what I usually do (boot once to generate keys, update the secrets, rebuild and switch remotely to the machine)
If bare metal recovery takes shorter time than reinstalling, then you should make bare metal backups. When bare metal updates, so should your backups.
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
You've missed the point. It doesn't matter that you can rebuild the system exactly, the example in question had little to do with how accurate the restores were, it had to do with the time to reinstall.
I too can entirely rebuild many systems simple from kickstarting a new one and applying an ansible playbook. Do I want to? No, that's too slow, and I might be doing this operation for tens or hundreds of systems.
We have multiple levels of backups. We use veeam for VM and physical backups of the entire system to a disk store that's also shipped off-site to an immutable store, but we also run an older system based on rsync which rsyncs the system to directory on a server (sans some data directories) that uses snapshots to keep space usage minimal for the purpose.
In many cases, if we want to restore we can literally just rsync data back to the host, whether one or a few files or most of the system.
I can restore many types of failures within seconds or single digit minutes at most, and the time required is mostly unattended time. Time to restore is often an important and overlooked metric, until you've been bitten by it and realized that an hour or more of work from a person to restore something doesn't scale well in some scenarios, and those are often the most important ones.
But it's not actually slower. When NixOS loads you get a list of all previous states to chose from. Booting to any one of them is just as fast as booting to the most recent one.
Restoring some backup via copying bits around is much slower because you have to wait for those bits to copy, whereas booting to previous configs is just rejiggering references to bits which are already in place and need no manipulation.
Immutability buys you quite a lot here, although some up-front discipline is necessary to take advantage of it, which might be a dealbreaker for some.
2 replies →
I feel that this made more sense in 1997. Especially in the context of Windows Server and whatever people ran on it.
If you think this is historical, you have a lot to learn about production servers in the wild! =)
Just as a not-insignificant portion of the world still relies on a handful of ancient AS/400 systems, a pretty big portion of society depends on the good running of hand-configured Windows Server.
this is from 1997. much easier to do this in 2026.
Is it possible to do that through Veracity? I don't want to use a non-Veracity based solution when Veracity already supports it.
Does Nix keeps old version indefinitely ?
Like, you can do the same with any distro and say Puppet or Ansible but only to within that distro packages, any 3rd party stuff (and people use more and more stuff that's 3rd party to the distro) might just have dead link for the download.
Also, plain restore from backup is still faster
It does, actually! Because only the package definitions are needed, and you can get those from the tarball of the relevant nixpkgs repo commit. And since building+installing is the same Nix command as downloading+installing, whether the binary caches still have the old package (or are even still up) is only the difference between a fast download and a slower build. This is different from most package managers, where trying to install anything old means spinning up a complex build environment and learning a bunch of new tools.
Of course, if even the original sources are no longer available, you'd have to change the broken URLs. And if the source has somehow been scrubbed from the Internet you're SOL
I typically use restores from backups for my personal computer as opportunities to clean things up.
Reverting to a snapshot is a better approach than using backups. My host machine is clean, as all my work is done in VMs.
Tell that to MinIO users, lol.
> The novice became suspicious and said: "Master, is all this 'Tao of Backup' stuff just so you can sell more copies of Veracity?"
> The master said: "Now you are truly enlightened."
Lol, I got suspicious of it being an Ad as soon as that showed up, at least it has the guts to acknowledge it
> The master said, "Do not despair, for yesterday I took one of your backup tapes and posted it to my brother in China. He will return it."
The master should not have done this, even by sea mail, for the very first law of backup is: you will only learn to make proper backups after you suffer a catastrophic loss of data.
Protecting the student from this is depriving him of the only true knowledge, the one that is not merely stored in the head, but burned into the skin.
I miss the World Wide Web, man...
Hard same.
Perusing small blogs over feeds any day of the week.
Social media cannibalized most of the interesting blogs, and 92% of people no longer have the patience to scroll more than 1/2 a page of text. =3
For reference: a comprehensive backup + security plan for individuals https://nau.github.io/triplesec/
> Store your Really Important Info in a Google Drive/iCloud synced VeraCrypt volume
There is no way I would entrust Google or Apple with keeping important data. Google being infamous for account issues with absolutely no support and Apple already borked my account in a way that makes my iPad unusable the moment I turn on backup sync.
> Google banned you
> Access a local copy of your Really Important Info storage data on one of your devices. You are fine.
2 replies →
I have neither a macbook nor an iphone
That is horribly written.
please extend for
* windows * android
* specific applications like
* signal * photos
and son on :D
Great site - and lessons are still valid.
How you can tell it's from 1997:
The master fell silent, but on his way out he pocketed a 20 Gigabyte corporate backup tape that was lying around.
That's about the right size for all the data that actually matters at a big corporation. Everything else is just marketing slop that people think is important.
That is just nonsense. Is your work experience perhaps only in marketing-oriented shops for you to think that?
Most models from architect offices of any size will be larger; many experimental results from applied physics research labs will be larger; any single project from a film production company of any size will be larger; any combined weekly security log at MSPs of any size will be larger; any industrial plant's daily production indicators will be larger; any energy company's hourly operational measurements will be larger.
1 reply →
I just launched a music backup tool and service.
Storage is Hetzner Storage Box which so far feels great. Cheap and supports SSH SFTP WebDAV, so you can use rsync or restic and browse artifacts over www.
It feels a little slow hosting in Germany but using from US. But I haven’t found a provider with the same cost / features.
Any alternatives folks use for backups?
https://deadca7.com/blog/backups
I've been using rsync.net for years.
https://rsync.net
NetDynamics shares some promo codes every so often, they have a data center in Utah as well as the EU. I wouldn't call it super low latency -- but backup generally doesn't need to be. I've used both at various times for a personal off site backup target.
But for commercial use, IDK, you'd want to dig into the details of their system. Not exactly S3 levels of resilience, but perhaps enough for some people, I leave it to others to judge.
Borg + Backblaze B2
Haven't seen that in ages and thought it had moved. 20GB tape containing all the corporate goodies. That makes me smile.
That should still be the case today.
Other than photos, music and videos, everything I have ever done in the last 40 years is 1.57Gb.
Media is where it got expensive.
I worked for a small manufacturing company some years back, and depending on what you're doing it's easy to rack up some huge file storage usage and big databases without being in the business of video, photography or music. The database that only ran our manufacturing, not inclusive of HR, accounting, etc., was about 1TB. The design files that went to the plant (most of which were just text, you could open them with Notepad) were another 3 or 4 TB worth. Maybe another 1TB or so for various collateral bits of data related to jobs, mostly PDFs.
The company has, I'm sure, probably tripled all that by now. In their case it compressed really well, but still.
All that to say, YMMV, significantly.
Corporate backups include entire VMs, not just files. So you are looking at something that is closer to 40TB than to 40GB.
The $/TiB storage cost currently creates weird choices in backup media... even making some older media formats economical again. Just finished a consumer storage option survey yesterday.
Even if you account for consumer drive+spare cost, a 8TB HDD is still the best JBOD option. For under several dozen 25GB projects, the BD-R disc drive is currently the cheapest media backup option (about 35% cheaper than HDD above 12TiB). Note 50G and 100G BD EOL media are actually equal to or a worse deal than HDD.
Also, HPE LTO9 18GB Tapes are currently around 5 to 15 times cheaper than HDD and SSD in terms of data storage cost, but the fixed cost of tape auto-loaders and drives don't always make sense until you have a significant amount of backup data. Tapes may also create a single point of failure in use... so incremental backups are a non-zero risk, and triple redundant hardware is required (old hardware gets hard to find.)
The classic joke is still true: "Fast, cheap, or good.. choose any two..."
Best of luck =3
I looked in to the bluray thing and concluded it just doesn't make sense at any level. You have to buy a bluray writer which will become increasingly rare, and then you are limited to 25GB disks where you have to factor in a chunk of that will go unused as each project won't fit on exactly 25gb.
For the effort you are expending, just getting a hard drive makes way more sense even if the $/gb is technically a little worse. The utility of hard drives is much better.
4 replies →
I've been down "LTO at home?" rabbit-hole... I nearly ended up buying a SAS HBA, until I realized that S3 Glacier is tape-as-a-service.
13 replies →
After reading the title, but before visiting the link, thought this was a backup of all of Terrance Taos work.
Not to be confused with the Zen of Backup…
A novice asked the master: “What must I do to learn the Zen of Backup?” The master whacked the novice in the head, grabbed the keyboard, and ran sudo rm -rf /. The novice smiled and left feeling enlightened.
After six years on my old build, last week I built a new computer. One of the things I invested in was treating the backup regime as the most important aspect of the new build. There's a piece of mind one gets from achieving the "seven heads"
This was a true joy to read and as pleasant as ads can come.
I felt old seeing this posted again, but I think I feel even older due to the lack of comments saying "I remember this"!
I never forgot seeing this back from Slashdot days.
It's how I learned to back up data.
Well. That, and data loss.
I assume that this was a [very good] ad for Veracity.
Yes, but (and? Because?) the advice is very good regardless of the software used.
Agreed. I wish more ads were like that. Note that the advice is almost three decades old, and still absolutely relevant. The classics are always a hit...
I followed the link, and see that Veracity is still around, almost 30 years later, but in a very different form from what it must have been, back then.
And the cautious reader asked:
/next /next /next ...
And they closed the tab and flagged the submission even though the actual advice was reasonable and the product no longer existed.
To be fair it's a plug from 1997 and the domain has since changed hands
A weird reaction. You're missing some very good advice, advice that is completely oblivious to the backup product... a now-defunct product which is only advertised by a now-defunct company, just the once (that's refreshing), very clearly and honestly (that's refreshing), at the very end of the slideshow.
I would expect the Tao of backup to loose all data every so often. What is success without failure?
Your fast and lose with the english language.
Information wants to be loose!
lose
Wow,.. this is so old that they still use the word "Tao" instead of "Dao". Still good advice, too.
Nice Epilogue
I like that the site is simple text (if not plain writing), but even in 1997 this multi-page layout would've a pain to read vs. a single page.
> But the master had already overwritten it with a copy of DOOM.
A master after my own heart.
I read all the steps, but not yet enlightened.
You have to write them first.
Now trying to think of a Unix joke involving write, read and tee that also makes sense for the backup topic ...
the backup that can be named is not the eternal backup
Loss of hard disk is a great way to declutter your system just saying
Obligatory link to the story about how Toy Story 2 got deleted.
https://www.youtube.com/watch?v=8dhp_20j0Ys
PS If you work in DevOps, SRE or Storage, this video may give you a panic attack
I accidentally let qwen wipe my 60tb nas the other day, and the tao of that is "well at least i dont have to stress about what im doing with my 30 years of rando .docs, facebook pics, games, ebooks and star trek episodes anymore"
c'est la vie, it was only 2tb filled and mostly games
Like giving a 6yo unix jr eng a crayon and root login. "you made them ALL apfs at the same time??" sorry dad. i can't believe people use this in prod, but i know i'm also not flagging perms my pi is ready to roll! gotta watch qwen3.6, they will be getting demoted.
Man, that sucks. I always feel incredible loss when just one single file is gone forever. Can't imagine what it's like to lose everything.
I spent lots of time thinking about backups. I've got files dating back to 1991 and even some from the 80s (but I don't care about those so didn't bother backing them up: when the last 5"1/4 shall stop reading, those will be gone and it's fine).
I've even got stuff like copies of early websites made by friends, websites long gone and I'm pretty sure I'm the only one to still have backups of those.
To me verifying the backup should be part of the backup procedure itself and so I did just that: my backup procedure does verify that the backups can be decrypted/unarchived and it then verifies the files inside the backup.
Now the thing is: you don't need to verify 100% of your backups all the time (as in I don't decrypt and verify the checksums of 100% of my backups all the time). You can do random sampling: over n backups, you can be reasonably sure at least x% of the files are correct.
So random sampling is part of my backup procedure.
As those are encrypted backups, the verification also ensures that the backups can be decrypted (would be too bad otherwise, wouldn't it?).
I also believe the backuping procedure should not be able to go wild and destroy data. So my backup procedure is done by a podman container that access the volume with all my data (the one that needs to be backed up) read-only.
> To believe in one's backups is one thing. To have to use them is another."
Yes, which is why a proper backup procedure verifies the backup it just created. The greenlight is only given to the backup that's just been made once it's been verified by my verification script.
Some stuff can be verified automatically: for example say you backup a Git repo: a strict, full, git fsck on the backup (that is: on the Git repo once pulled out of the backup, before greenlighting it) works fine.
For other things I've got my own verifications.
> These hundreds of corrupted files have been flowing through my backup system. Now I do not know which files are clean and which are not.
Which is why many of my files, which I know aren't supposed to ever change anymore, have a partial checksum added as part of the filename.
I don't have:
but:
where 7e228491a0 are the first 40 bits of the Blake3 checksum of the file.
Then it become very complicated for "something" (bit rot or malicious) to silently corrupt my stuff for the backuping procedure (which only has read-only access to data, remember) goes crazy bonkers and warns me as soon as two identically named files (one on the last backup and the current version) have an identical name but different content and one doesn't match the checksum (this is cause for a "stop the world" and immediate enquiry: and, yup, it already allowed me to catch a corrupted file).
I've moreover got a sheet of paper, laminated, that explains how to access the backups and that explanation is saved in many places (safe at the bank, safe at my brother's house in another country, etc.): should say my place burn with me inside, but my wife be safe, she'd be able to access the backups too.
As for my main data, it's RAID on ZFS on a machine with ECC and that's where the backups are made (and automated) from.
Proper backup procedure is not hard: it just takes some time to set up properly, once. Then it's done for a lifetime.
Yeah... I'm going to have to incorporate the checksum-in-the-name idea into my photography workflow.
[flagged]
[flagged]
HTTPS cert is invalid
there is no cert, it's http://
You have learned another lesson: that a TLS/SSL cert is not necessary to achieve enlightenment! /s
It's 2026. Use TLS everywhere.