Comment by FrostKiwi

Have my server infrastructure on NixOS. Huge boost in productivity and stability, would never go back to standard linux. But man if something breaks it's a nuke going off. Sry for the long post, but thought I share my experience:

Mass storage on a big encrypted RaidZ array of spinning rust, no issues. Bootloader, /boot, Encrypted Root and Databases on Mirrored NVME Drives. And man is that a nailbiter on each update. Setup my drives during 22.11 following NixOS Root on ZFS instructions [1], which were amended following reports of systems becoming unbootable [2] and mostly removed later [3].

Besides initially being a broken setup [4] with an increasing amount of mounts each update stalling any system writes and causing updates to fail, it became a well running machine after that was fixed. Then during the 24.05 update and with no config change, the system became unbootable [5]. After a tough recovery [6] I never figured out how to do mirrored bootloaders again, switched to a single bootloader setup. To this day I have interactions I don't understand and am trying to fix [7], which sometimes causes updates to knock services offline due to the `nixos-rebuild switch` process stopping services, going to update the bootloader, failing due to missing mounts and exiting with services being offline, prompting manual intervention.

[1] https://github.com/openzfs/openzfs-docs/blob/1211e98faf1f37a...

[2] https://github.com/openzfs/openzfs-docs/commit/1211e98faf1f3...

[3] https://github.com/openzfs/openzfs-docs/commit/4fb5fb694f44c...

[4] https://github.com/NixOS/nixpkgs/issues/214871

[5] https://github.com/openzfs/openzfs-docs/issues/531

[6] https://github.com/openzfs/openzfs-docs/issues/531#issuecomm...

[7] https://discord.com/channels/568306982717751326/132854109967...