For root filesystems I am recently favoring EROFS which is read only, fast, and can run from ram.
On workstations I install almost all software to ~/.local as the home partition is writable. I also put all my systemd user units there, so I can run any binaries I compile as a system service to survive reboots as I like all without root.
The system root partition should contain a kernel, init system, and run any essential shared services unprivileged and fully/mostly stateless. Given that, I like to compile them all into a single UEFI uki image that contains efi shim, kernel, init all in a single binary that lives in the FAT32 UEFI partition.
The only reason to touch it is when you need to update your init system or kernel, which were always going to require a reboot anyway unless you get really creative with kexec. In an ideal situation the uki bundle is so generic that it is built deterministically in multiple locations and signed with a secure boot key. Then you can just straight up allow users to write to the boot partition, knowing any unsigned image that is not newer than the current one will be detected and also not allow access to the encrypted disk. The permission for that one update path can and should be external, and the result of a deterministic build matching a known hash, so you can prove it is not compromised. This could of course be automated by a distro with a UEFI wrapper or coreboot so users with no desire to customize their kernels do not have to think about it.
I am pursuing these ideas in stagex, first for secure enclaves and servers where we need it most, then for desktop. Until then Qubes is the least bad option.
It's cool that you're experimenting, but if all the apps that matter are installed to ~/.local then you're just removing the need to obtain root at all for most attacker priorities.
A bit more than experimenting. I have designed several security hardened linux operating systems for major financial institutions.
There are many well established patterns for further segmenting user workloads, once far removed from real system root which is there is no good reason to expose. In my case each of those unprivileged apps runs in a dedicated virtual machine because I use QubesOS, though for many use cases gvisor or even containers (user namespaces) would get the job done with less overhead.
gvisor in particular helps you further reduce attack surface for root. It is possible to have your baremetal kernel not even have network support compiled in, and delegate all networking entirely to user-space inside gvisor.
stickynotememo: "How would you install packages (or update the kernel)?"
You: «You'd use sudo or su (of course) but I'm going to distract you from that with an advertisement for the work I'm doing on my personal project!» [0]
But, perhaps I misunderstood what you wrote. So... I'll ask stickynotememo's question in a different way:
How would a user of your system update the Systemd collection of software to apply a critical security update?
If the answer is something like "I'd rebuild the read-only root partition with the new code.", then I ask:
1) How does the data in that root partition get rebuilt? If it's on another computer, how does one control access to the root-partition-rebuild computer?
2) How does one instruct the user's computer to use that newly-rebuilt root partition? How does one control access to the system that permits one to load a new root partition and/or change which root partition to use?
3) How does a user recover when an update fails or is faulty in a way that your smoke tests didn't catch?
Please keep your answers concise.
[0] Personal projects are great, and I strongly encourage them. However, the way you've answered is what a greybeard would describe as "pulling a fast one". Pulling a fast one does not earn trust.
I am not convinced you processed what I wrote, but I will attempt to answer your questions anyway because they will aid me in writing docs later.
> 1) How does the data in that root partition get rebuilt? If it's on another computer, how does one control access to the root-partition-rebuild computer?
If they use an official one published by their distro, then it is likely sufficient that it is signed by a key in a multi-party-controlled remotely attestable secure enclave controlled by the distro team, which only does so in exchange for multiple signatures from independently controlled secure enclaves that produce deterministic artifacts. The key could be backed up and restored across enclave updates using shamir secret sharing so no single engineer gets the key but they can cooperate to provide it to an enclave.
Of course this scheme to avoid trust in any single human or machine in the linux distribution supply chain only works if you happen to have a deterministic full source bootstrapped linux distribution that has strong multi-party security controls (like stagex, my "personal project", protecting hundreds of billions of dollars in fintech infrastructure right now)
> 2) How does one instruct the user's computer to use that newly-rebuilt root partition? How does one control access to the system that permits one to load a new root partition and/or change which root partition to use?
If the high trust supply chain signed UKI builds are a thing, per question one, then life gets easy here.
On first install a user would be forced to enable secure boot enrolling the linux distro signing keys, and the installer would encrypt their hard drive to that secure boot state via TPM PCRs.
Now, even though the user-accessible portion of the OS is entirely unprivileged, the user can be trusted to download new signed UKI images and place them in an update search path. Now on reboot the update will be seen, verified newer, and could be kexeced to. If it boots successfully past the uki image, a hook will cause it to move the old image to a backup file and replace with the new one, then reboot properly. Secure boot sigs pass, image is newer than old image, TPM policies pass, and disk can be decrypted. All automated.
> 3) How does a user recover when an update fails or is faulty in a way that your smoke tests didn't catch?
If the kexec test fails to boot to the hook and drops a crashlog, then the original still-unmodified UKI image stays in place, will see the crashlog, and then boot normally and warn the user about the failed update and offer to submit an Issue.
None of this is novel, though knowledge of it seems sadly rare. All can be done right now with existing FOSS, and many embedded Linux devices and servers use patterns like these. Especially in high security environments like fintech. Just configuration, admittedly a lot of it, which is what I am packaging and simplifying right now in stagex so it can be an opinionated default.
For root filesystems I am recently favoring EROFS which is read only, fast, and can run from ram.
On workstations I install almost all software to ~/.local as the home partition is writable. I also put all my systemd user units there, so I can run any binaries I compile as a system service to survive reboots as I like all without root.
The system root partition should contain a kernel, init system, and run any essential shared services unprivileged and fully/mostly stateless. Given that, I like to compile them all into a single UEFI uki image that contains efi shim, kernel, init all in a single binary that lives in the FAT32 UEFI partition.
The only reason to touch it is when you need to update your init system or kernel, which were always going to require a reboot anyway unless you get really creative with kexec. In an ideal situation the uki bundle is so generic that it is built deterministically in multiple locations and signed with a secure boot key. Then you can just straight up allow users to write to the boot partition, knowing any unsigned image that is not newer than the current one will be detected and also not allow access to the encrypted disk. The permission for that one update path can and should be external, and the result of a deterministic build matching a known hash, so you can prove it is not compromised. This could of course be automated by a distro with a UEFI wrapper or coreboot so users with no desire to customize their kernels do not have to think about it.
I am pursuing these ideas in stagex, first for secure enclaves and servers where we need it most, then for desktop. Until then Qubes is the least bad option.
It's cool that you're experimenting, but if all the apps that matter are installed to ~/.local then you're just removing the need to obtain root at all for most attacker priorities.
A bit more than experimenting. I have designed several security hardened linux operating systems for major financial institutions.
There are many well established patterns for further segmenting user workloads, once far removed from real system root which is there is no good reason to expose. In my case each of those unprivileged apps runs in a dedicated virtual machine because I use QubesOS, though for many use cases gvisor or even containers (user namespaces) would get the job done with less overhead.
gvisor in particular helps you further reduce attack surface for root. It is possible to have your baremetal kernel not even have network support compiled in, and delegate all networking entirely to user-space inside gvisor.
To summarize:
stickynotememo: "How would you install packages (or update the kernel)?"
You: «You'd use sudo or su (of course) but I'm going to distract you from that with an advertisement for the work I'm doing on my personal project!» [0]
But, perhaps I misunderstood what you wrote. So... I'll ask stickynotememo's question in a different way:
If the answer is something like "I'd rebuild the read-only root partition with the new code.", then I ask:
Please keep your answers concise.
[0] Personal projects are great, and I strongly encourage them. However, the way you've answered is what a greybeard would describe as "pulling a fast one". Pulling a fast one does not earn trust.
I am not convinced you processed what I wrote, but I will attempt to answer your questions anyway because they will aid me in writing docs later.
> 1) How does the data in that root partition get rebuilt? If it's on another computer, how does one control access to the root-partition-rebuild computer?
If they use an official one published by their distro, then it is likely sufficient that it is signed by a key in a multi-party-controlled remotely attestable secure enclave controlled by the distro team, which only does so in exchange for multiple signatures from independently controlled secure enclaves that produce deterministic artifacts. The key could be backed up and restored across enclave updates using shamir secret sharing so no single engineer gets the key but they can cooperate to provide it to an enclave.
Of course this scheme to avoid trust in any single human or machine in the linux distribution supply chain only works if you happen to have a deterministic full source bootstrapped linux distribution that has strong multi-party security controls (like stagex, my "personal project", protecting hundreds of billions of dollars in fintech infrastructure right now)
> 2) How does one instruct the user's computer to use that newly-rebuilt root partition? How does one control access to the system that permits one to load a new root partition and/or change which root partition to use?
If the high trust supply chain signed UKI builds are a thing, per question one, then life gets easy here.
On first install a user would be forced to enable secure boot enrolling the linux distro signing keys, and the installer would encrypt their hard drive to that secure boot state via TPM PCRs.
Now, even though the user-accessible portion of the OS is entirely unprivileged, the user can be trusted to download new signed UKI images and place them in an update search path. Now on reboot the update will be seen, verified newer, and could be kexeced to. If it boots successfully past the uki image, a hook will cause it to move the old image to a backup file and replace with the new one, then reboot properly. Secure boot sigs pass, image is newer than old image, TPM policies pass, and disk can be decrypted. All automated.
> 3) How does a user recover when an update fails or is faulty in a way that your smoke tests didn't catch?
If the kexec test fails to boot to the hook and drops a crashlog, then the original still-unmodified UKI image stays in place, will see the crashlog, and then boot normally and warn the user about the failed update and offer to submit an Issue.
None of this is novel, though knowledge of it seems sadly rare. All can be done right now with existing FOSS, and many embedded Linux devices and servers use patterns like these. Especially in high security environments like fintech. Just configuration, admittedly a lot of it, which is what I am packaging and simplifying right now in stagex so it can be an opinionated default.
A related "Personal project": https://distrust.co/blog/enclaveos.html