Comment by k__

13 hours ago

How do you deploy NixOS to your servers?

I started a project with nixos-anywhere, deploy-rs, and compose2nix. However, I struggle a bit with secret management.

9 comments

k__

Reply
majoe  8 hours ago

I struggled with remote deployment + secret management, too. Like a lot of folks, my nix-config grew over the years as I added secrets management, user management etc ad hoc.

I recently found clan.nix [1] and am quite pleased. It's kind of a framework for writing nixos configurations with a focus on multiple devices. It bundles secrets management and remote deployment into a convenient CLI.

It has the concept of "services", which are e.g. used for user management and VPNs. Services define roles, which can be assigned to machines, e.g. the wireguard service has a controller and a peer role. That feels like the right abstraction and it was very easy to set up a VPN with zerotier like that, something I struggled doing myself in the past.

It's a rather young project, but I converted my nix-config repo after a short evaluation phase to use clan. It's worth taking a look for sure.

[1]: https://clan.lol/

  • c0balt  8 hours ago

    I would strongly recommend sops-nix[0]. Pair this with ssh-to-age/ssh-to-gpg for the keys for each server. We are using this at $work for multiple servers, one notable advantages is that it works in teams (for multiple people) and git (and also gitops).

    For remote installations nixos-anywhere is great. deploy-rs or colemna is fine, nixos-rebuild with `--target-host` is also working well for us however.

    [0]: https://github.com/Mic92/sops-nix/issues

drdaeman  11 hours ago

There are plenty of options: nix-sops, or nix-age, or whatever you would like - past the overall idea the implementation details are purely a matter of taste how you fancy things to be. Key idea is to have encrypted secrets in the store, decrypted at runtime using machine-specific credentials (host SSH keys are a typical option to repurpose, or you can set up something else). For local management you also encrypt to the “developer” keys (under the hood data is symmetrically encrypted with a random key and that key is encrypted to every key you have - machines and humans).

Alternatively, you can set up a secrets service (like a password manager/vault) and source from that. Difference is where the secrets live (encrypted store or networked service, with all the consequences of every approach), commonality is that they’re fetched at runtime, before your programs start.

I’m currently using deploy-rs, but if I’d redo my stuff (and the only reason I don’t is that I’m pretty much overwhelmed by life) I’d probably go with plain vanilla nixos-rebuild --target-host and skip any additional layers (that introduce extra complexity and fragility).

Just a crude and somewhat haphazard summary but hope it helps.

  • VTimofeenko  10 hours ago

    I'd recommend sticking with deploy-rs. Saved me quite a few hours through its magic rollback which aborted an upgrade that borked VPN.

  • diarrhea  10 hours ago

    I have been using nixos-rebuild with target host and it has been totally fine.

    The only thing I have not solved is password-protected sudo on the target host. I deploy using a dedicated user, which has passwordless sudo set up to work. Seems like a necessary evil.

    • JamesSwift  4 hours ago

      I do this to remote deploy and it works fine even from my mac

      > nix run nixpkgs#nixos-rebuild -- switch --flake .#my-flake-target --target-host nixos@$192.168.x.x --sudo --ask-sudo-password --no-reexec

    • matrss  10 hours ago

      > I deploy using a dedicated user, which has passwordless sudo set up to work.

      IMO there is no point in doing that over just using root, maybe unless you have multiple administrators and do it for audit purposes.

      Anyway, what you can do is have a dedicated deployment key that is only allowed to execute a subset of commands (via the command= option in authorized_keys). I've used it to only allow starting the nixos-upgrade.service (and some other not necessarily required things), which then pulls updates from a predefined location.

embedding-shape  9 hours ago

I seemingly went what I consider simple, others might consider barebones: I have a Makefile, define all the hosts there, and then use scp to copy over all files into the right hosts, and then run `nixos-rebuild switch` on them once copied over. Some have extra "restart-service-X" as some services have their configuration managed outside NixOS (but still in my source repository) so those as well are scp'd into the right place, then the service restarted after the nixos-rebuild switch. Otherwise it's all .nix files and service-specific configuration files.