Comment by dbdr
17 hours ago
The novice said: "I will save my working files, but not my system and application files, as they can be always be reinstalled from their distribution disks."
The master made no reply.
The next day, the novice's disk crashed. Three days later, the novice was still reinstalling software.
If that's your situation, it might be worth backing up applications. However, there are better solutions that make this irrelevant. For instance, using NixOS, you can literally back up a single text file describing your system setup, and you can recreate the system from it automatically in minutes, guaranteed to be bit for bit identical.
In my experience that works in box like 80% of the time. Which is great! But a backup is great in case it doesn’t.
If you are wondering ‘How are you getting that much failure’, it’s because I like to freeze older versions and they often decay for one reason or another (can’t get files for example)
Backup the distribution files via mirror/proxy-mirror?
I think the point was a single file for bit by bit replication. Managing infrastructure isn't good ROI here.
1 reply →
I mean I could but I find a full drive backup isn’t that much bigger than a more selective backup and whenever I try to back up part of a drive I never grab the right bits.
My strategy for Gentoo, more or less:
/etc/ - all modified configuration files, including
/etc/portage/ - make.conf, package configuration (package.accept_keywords, package.use, and friends)
/var/lib/portage/ - world, and world sets. basically everything portage needs to rebuild the system
/var/db/repos/ - notably local and/or self-owned package repos
/home/<username>/ - no explanation needed
/root/ - no explanation needed
This should contain enough to pretty much rebuild a Gentoo system from a clean downloadable Stage onwards without (too much) intervention.
It's not guaranteed to be bit identical (the why of it is a can of worms), but it's identical enough for all practical purposes.
At the same time it is significantly more "hardware-change" resilient, since the same backed up config file (minus perhaps some very system specific lines that you may or may not have) can bring up a functionally equivalent working system on quite distinct other hardware as well.
If your NixOS setup is not of hello-world kind, you typically have sops in it (or another secret management tool) and want to backup your age keys as well. They're necessarily separated from the main config tree so they don't end up leaking to the globally accessible nix store, and in the most typical case they're located in a 600 file outside the git repo. Forgetting to back it up is a classic NixOS gotcha and also a generalized version of this koan.
Wouldn't you just re-encrypt the secrets with a new host key? That's what I usually do (boot once to generate keys, update the secrets, rebuild and switch remotely to the machine)
If bare metal recovery takes shorter time than reinstalling, then you should make bare metal backups. When bare metal updates, so should your backups.
If you're using virtual machines, make sure to backup the hypervisor config and setup as well. You never know which part of the infrastructure starts to fail first.
I feel that this made more sense in 1997. Especially in the context of Windows Server and whatever people ran on it.
If you think this is historical, you have a lot to learn about production servers in the wild! =)
Just as a not-insignificant portion of the world still relies on a handful of ancient AS/400 systems, a pretty big portion of society depends on the good running of hand-configured Windows Server.
You've missed the point. It doesn't matter that you can rebuild the system exactly, the example in question had little to do with how accurate the restores were, it had to do with the time to reinstall.
I too can entirely rebuild many systems simple from kickstarting a new one and applying an ansible playbook. Do I want to? No, that's too slow, and I might be doing this operation for tens or hundreds of systems.
We have multiple levels of backups. We use veeam for VM and physical backups of the entire system to a disk store that's also shipped off-site to an immutable store, but we also run an older system based on rsync which rsyncs the system to directory on a server (sans some data directories) that uses snapshots to keep space usage minimal for the purpose.
In many cases, if we want to restore we can literally just rsync data back to the host, whether one or a few files or most of the system.
I can restore many types of failures within seconds or single digit minutes at most, and the time required is mostly unattended time. Time to restore is often an important and overlooked metric, until you've been bitten by it and realized that an hour or more of work from a person to restore something doesn't scale well in some scenarios, and those are often the most important ones.
But it's not actually slower. When NixOS loads you get a list of all previous states to chose from. Booting to any one of them is just as fast as booting to the most recent one.
Restoring some backup via copying bits around is much slower because you have to wait for those bits to copy, whereas booting to previous configs is just rejiggering references to bits which are already in place and need no manipulation.
Immutability buys you quite a lot here, although some up-front discipline is necessary to take advantage of it, which might be a dealbreaker for some.
That's useful if the system is still up and you can trust the state information, but when you have a disk failure, or there's enough system state corruption that you don't want to rely on that state tracking, you're then going to need something else.
Also, are you pinning your package versions in your config? For every single case of that, is the original package still available? Were they coming from remote sources which may not keep prior versions always available?
Did you build anything from scratch? Did you supply a source or binary archive to install from? Is that source still available, of the exact same version?
There are numerous things you need to worry about when rebuilding from what the state should be based on a description of that state (which may include arbitrary actions), as opposed to restoring from a recorded state that includes everything needed in the recorded data.
The more you can record the better. Should you also record nix configuration state and use that to restore when it makes sense? Sure! If it's simple and easy and doesn't use an inordinate amount of space, why wouldn't you? Just don't confuse nice to have with sufficient based on the specific requirements and goals your backup system needs to meet.
We have entire system images in an immutable remote store, but that's not what we go to first, it's what we go to when our quicker and easier systems are insufficient. If we were running NixOS, we'd probably keep everything we're doing now, and then also just add the NixOS state as a separate thing that's backed up and saved on every change (and in fact, along these lines, we also run etckeeper on every host and track all changes to /etc with daily commits if something is changes but also an automatic commit at the beginning and end of every ansible run we do that applies our host configuration).
More is better. If I want to see configuration changes, I can usually just look in etckeeper on the box, if I want to restore a tree to the exact state it was at a specific point that a backup was made through rsync, I can do that easily enough (or just navigate the directory tree on the backup server, if I just want to see what it was, or diff between different states at different times). If I need to restore the whole system image from a known good backup, I can use Veeam and the local repository for that. If the local repository is dead, I can use the replicated off-site repository. If an attacker has destroyed both those, I go to the immutable off-site backups, which I'm probably doing to use after rebuilding my backup infra from scratch to restore from. Each of these protects from slightly different scenarios, and offers different trade-offs for security and integrity and ease of use.
1 reply →
this is from 1997. much easier to do this in 2026.
Is it possible to do that through Veracity? I don't want to use a non-Veracity based solution when Veracity already supports it.
Does Nix keeps old version indefinitely ?
Like, you can do the same with any distro and say Puppet or Ansible but only to within that distro packages, any 3rd party stuff (and people use more and more stuff that's 3rd party to the distro) might just have dead link for the download.
Also, plain restore from backup is still faster
It does, actually! Because only the package definitions are needed, and you can get those from the tarball of the relevant nixpkgs repo commit. And since building+installing is the same Nix command as downloading+installing, whether the binary caches still have the old package (or are even still up) is only the difference between a fast download and a slower build. This is different from most package managers, where trying to install anything old means spinning up a complex build environment and learning a bunch of new tools.
Of course, if even the original sources are no longer available, you'd have to change the broken URLs. And if the source has somehow been scrubbed from the Internet you're SOL
I typically use restores from backups for my personal computer as opportunities to clean things up.
Reverting to a snapshot is a better approach than using backups. My host machine is clean, as all my work is done in VMs.
Tell that to MinIO users, lol.