Comment by abejora

I agree with this sentiment! For our Abejora timesheet SaaS, setting up backups was one of the harder parts. We had to change direction a few times while implementing it.

We finally got a nice setup with CloudNativePG + Barman. This allows for point-in-time restores, but there were a lot of lessons to learn along the way.

- The various types of (database) backups (logical, binary, onsite, offsite, snapshots, write-ahead log...) in combination with the various types of data (database, files, cluster configuration...)

- In our earlier approaches, we tried to preserve the old database volume if it was not corrupt, and use that in our restore. This caused so many complications, because you are fighting the recommended approach. So now, when we need to restore, we always restore from backups and the 'live volume' is dropped.

- For a restore, we just spin up a completely new Kubernetes cluster, instead of trying to restore in-cluster. This is a lot easier.

- Many object stores allow for retention periods, which you can put to good use to prevent malicious or accidental removal of backups. HOWEVER, not all of them are really 'locked'. In some services, you can still override the lock with a forced delete; in others, you can still remove the project holding the storage buckets, which will delete the buckets, and so on... so test those things, instead of just blindly depending on a 'retention period' claim.

- We now automatically run a scheduled restore with verifications on a weekly basis. This requirement does shape your environment, so keep that in mind! There is also the question of how you can reliably and automatically verify that the restore restored the latest data (of a live prod environment). Various solutions exist here, but most are not very elegant!

Honestly, this is only worth it if you are already handling sufficient volume. If you are just starting out, then the easier approach is to just go with a hosted database, which will handle backups and point-in-time restores for you.