Comment by pojntfx

> is currently converging on a different proposal by systemd developers that's standardized by the UAPI Group

We're working in this space with Project Bluefin: https://github.com/projectbluefin/dakota

Both approaches are indeed competitive, but you can also leverage both to achieve the same thing. We're experimenting with a pure ddi Bluefin, a buildstream/GNOMEOS one that spits out a bootc image, as well as a Bluefin that is just a systemd-sysext on top of GNOME OS. Chef's choice!

There will be many ways to slice this problem -- my opinion is that in the end it will be how you design the infrastructure to make these and not the artifacts themselves.

We already have CentOS/Fedora builds alongside these, long term we'll see which ones end up being the most efficient. Buildstream is a tool which people should look at in this space too: https://buildstream.build/index.html

sysexts are indeed a very interesting feature, though it really only complements some other whole-system solution since it can only affect files under a non-root folder location.

I'm struggling to see how sysupdate is really equivalent to bootc or ostree though. Sysupdate is just the sw-update tool from Yocto that's been around for 10+ years with a little more syntax sugar, which itself was just a common shared implementation of what all embedded systems had been rolling-thier-own of for almost 20 years before that. It says it requires an A/B/.../N partitioning scheme, which is exactly what bootc/ostree/etc is designing to avoid.

If you don't use the whole disks update thing from sysupdate, then instead you're just talking about a transactional package manager that is still in its infancy and hasn't addressed the many gotcha and corner cases that the dozens of more mature package managers have. It's not actually "transactional" in the sense of undo for example, it's "transactional" only in that you won't get partial install, which hasn't been a problem with any existing package managers for almost 40 years. All thier listed things you can list together for a "transaction" association are either things that are already linked via existing package maager packages, or are only useful for embedded systems.

I'm not saying sysupdate isn't useful, upper end of embedded design is pushing into the space where systemd is standard now so it could be useful for those devices, but it's not really equivalent at all to bootc/ostree, and doesn't really have amt applicability outside initial system imaging from a live disk, or embedded devices.

Ironically the first implementation of ostree required an HTTPS server to serve the ostree commits, allowing a much smaller subset of what's needed to be transferred. However that became an adoption hurdle since it required unique infrastructure. Ostree switched to using containers because zstd allows compressed chunks now rather than the old all-or-nothing image layers, and the existing widespread container image registry infrastructure could be reused without modification. And both utilize layers for their construction so there were possible benefits there (that never really materialized, but are still available to pursue).

Typo: (CoreOS and Fedora Silverblue) are the bleeding edge of immutable distros. Those mentioned are just users.

> the bleeding edge of immutable Linux distros (GNOME OS, KDE Linux)

These are words but they don't make sense.

From https://uapi-group.org/ :

> Contributing members include people from Ubuntu Core, Debian, GNOME OS, Fedora CoreOS, Endless OS, Arch Linux, SUSE, Flatcar, systemd, image-builder/osbuild, mkosi, tpm2-software, System Transparency, buildstream, BTRFS, bootc, composefs, (rpm-)ostree, Microsoft, Amazon, and Meta.

Note systemd, (rpm-)ostree and bootc.

My understanding is that uapi is another initiative but not completely separated from bootc and ostree. Maybe complementary.

I often see bootc and/or buildstream uncritically presented as the future of the linux desktop, and find it somewhat surprising because, in my experience, they are both more complex _and_ less capable than nixos.

To elaborate, I will separate linux operating systems into three categories:

1. traditional mutable package management (debian, fedora, arch, etc) - Bad model for obvious reasons, I won't get into this.

2. immutable images (embedded deployments, android, chromeos) - A build system (yocto, nix, buildroot, buildstream) procudes a disk image. System boots the disk image and handles updates using A/B root partitions with `systemd-sysupdate` or similar.

3. store-based atomic (nixos, ublue) - The system keeps a mutable store of hashed objects (/nix/store, /var/lib/containers/storage, /ostree/repo) and boots a specific system generation. Updates add new objects to the store, which must be automatically cleaned, and create a new systemd generation to boot into.

In the case of categories 2 and 3, a build system of some kind is used to produce the image or packages that are turned into the new system generation.

The bootc project, which falls into category 3, attempts to use the standardized and highly adopted OCI image format (layered filesystem changes stored in a content addressed store) as a medium for distributing linux systems. The major limitation here is that these systems are very complicated to build and extend. While it may not seem that way compared to nixos if you have prior experience writing dockerfiles, configuring your system with imperative statements that build on previous state is _really_ tedious. For example, you can't just re-use work with multiple `FROM` statements in the same layer, so you instead need to copy files between images. This is incredibly jank, look at [the docs](https://blue-build.org/how-to/minimal-setup/) for bluebuild's module system. Additionally, for a motivated user to change their system internals, they need to make the jump to hosting it with CI and pulling the images.

As jcastro mentioned elsewhere in this thread, there is work in the ublue project, which focuses on bootc images, to instead build their systems from source using buildstream, the same way GNOME OS works. The idea is that this is no longer a "distribution" and doesn't have "packages" anymore, since the entire system is built from source into an image that can be updated to atomically. While this model is simpler and way less jank than assembling your OS in a dockerfile, the model of separating packaging from distribution makes things harder for users for no benefit over the alternative.

To elaborate: Buildstream is a build system created by gnome developers that works as follows:

- Various element are defined in yaml files with their dependency and build steps

- Buildstream forms a tree of build elements to evaluate

- Each element is evaluated in a sandbox with access to only its dependencies and the output is placed in a content-addressed store for caching

If you have experience with nix, you may recognize that this is almost the same way nix works, with the difference that buildstream dependencies are mounted to regular directories (e.g. /usr) in the build sandbox instead of directly dealt with as /nix/store objects, and that buildstream is much harder to extend. A nix build results in artifacts that live in the nix store and are symlinked to other paths in the nix store, while a buildstream build results in artifacts that are compatible with a more traditional directory structure.

The idea of GNOME OS being "distroless" and not having packages is misleading, as it does have packages. Only, the packages exist exclusively during the build process. In order for a user to modify their system, they must add a rule to the buildstream definitions their system is built from and rebuild the entire thing from source to generate a new system image, which is unnecessarily burdensome. This is because the content-addressed artifact store (buildstream's cache) that exists when building has no relationship to the content-addressed artifact store on the deployed system (ostree or oci storage). This is pointless indirection with no clear benefit. By not separating building from distribution, nixos (a project using the nix dsl to build a linux system) achieves the same benefits of this model without any of the drawbacks. Users can modify how their system works, use caches, add their own packages, and share/integrate these modifications freely without building their entire os again from source every time.

To put this power in perspective, adding a nginx server to your system on nixos amounts to adding to your nix configuration:

services.nginx.enable = true;

and then rebuilding, switching to the new system generation.

Imagine how painful it would be to do this via a dockerfile (based off a different image) or buildstream definitions...

Finally, I'd like to clarify that nixos is not perfect. There are many areas of that need improvement (documentation, evaluation speed, evaluation caching, ifd, error messages, etc). However, I believe nixos is a fundamentally better and simpler model than the one being pushed by a lot of the linux desktop ecosystem at the moment. I believe a lot of the work on infrastructure like bootc and buildstream would be better focused on nix/nixos, or at least would benefit from learning from them.

> bootc and OSTree are both very neat

May I rephrase that?

bootc and OStree are both Cthulhoid nightmare horrors that only exist because of corporate politics, but the leading edge...

GNOME OS and KDE Linux are both specialized distros that primarily exist to test GNOME and KDE. They aren't for general users, and both web sites warn you not to rely on them. And they impose limitations on your ability to install arbitrary 3rd party software, whereas Fedora Atomic Desktop lets you customize the system without such limitations. Fedora Atomic lets me install arbitrary RPMs into the base system.

"quite a bit easier to extend" sounds good to me, but the "easier" here refers to the internal system implementation details? I am an end user, not a Linux distro system architect, and I care more about the user experience. I will be interested in test driving a general purpose OS based on this technology, whenever that happens in the future. Since Red Hat is involved in the UAPI project, perhaps Fedora Atomic Desktop will migrate to this technology in the future?