I work in the embedded space. I'd absolutely love to have a tool to build immutable, signed distro images which I can push to devices with a/b style updates. I imagine you can do this with mkosi, but it doesn't quite feel like the intended use case.
Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
Right now the industry standard tooling for building truly immutable images for embedded devices is Yocto. It works well, but it's incredibly complicated and has ridiculous build times because it insists on building every single package from source. It's utter madness that building a Linux image for a common ARM64 processor requires me to compile the Rust compiler myself.
You don't have to build ParticleOS images on the machine itself, it's perfectly possible to build them offline somewhere else and download them from the target machine when doing an update with systemd-sysupdate. It's just that we haven't quite gotten to ironing out all the details there. We're adding support to OBS (OpenSUSE Build System) to build ParticleOS images and will eventually start publishing prebuilt images there which can then be consumed by systemd-sysupdate.
For the embedded space you'd just build the ParticleOS images on your own build system, publish them somewhere and consume them with systemd-sysupdate, doesn't have to be OBS of course.
But we don't do stuff like only downloading diffs with systemd-sysupdate and such yet, so your milleage may vary.
That's interesting. I've been looking for an angle on embedded Linux software updates for my deployment tool (currently limited to updating desktop and server apps only).
When you say push, do you literally mean push or do you mean the devices would still pull updates on their own schedule where you get to control rollout percentages, channels, etc centrally? Mostly devices have to be assumed to be behind NAT, right?
What I'm thinking here is maybe it'd be useful to have a mode in Conveyor [1] that builds a standalone ARM Linux that can boot and pull updates for everything including the app, but which coordinates with the userspace app to decide when it's time to restart and for which the build to update to is controlled by server-supplied metadata so you can divide the fleet by tag, etc.
The problem with immutable images is you need a complimentary config management tool to manage them at scale. That's why we built Etcha[0] (config management for 1000+ devices) and then built an immutable OS (EtchaOS) around it--they're meant to work together instead of "cloud-init is good enough". cloud-init is terrible at scale, especially outside of public cloud.
> Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
This is how it always has been on enterprise network devices like switches, routers, etc. It’s pretty trivial to automate updates.
I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
>Sometimes ParticleOS adopts systemd features as soon as they get merged into systemd without waiting for an official release. As a result it's recommended to build systemd from source when building ParticleOS to make sure all required features are supported
> I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
You don't have to, we build packages and publish repositories from latest main, and the particle configs can use those by enabling the 'obs' build profile
As in: "you've tried alternatives and you really think that systemd is superior", or as in "you don't see why people complain about systemd and its philosophy"?
Genuinely interested, because usually people I hear say that they don't see the problem with systemd don't really have experience with the alternatives. Which makes sense: people who like being able to use alternatives tend to get screwed by systemd :-).
Edit: for those downvoting, this is a genuine question. I mostly hear opinions from people who hate systemd, and those who don't generally just don't really care: "I've always used systemd and I haven't had issues". But the latter group wouldn't call themselves "fans of systemd".
Systemd makes me sad because 99% of what I like about it could have been implemented in a modular fashion, but instead it's mostly all-or-nothing. I still use OpenRC on servers, but I have to implement or integrate a lot of the missing features, like all the cgroup/namespace stuff, logging, etc.
I've used Solaris SMF back in the day, which is what systemd largely copied, but badly.
The major difference is, in SMF you specify services, their dependencies and whatever else you need. Then SMF looks at the whole desired system, computes the dependency tree and boot order, validates everything and writes a boot plan that can be executed in a static and deterministic fashion. You can look at the plan, execute that exact same plan each boot. You therefore know that if the system came up once using that plan it will again.
Whereas systemd does everything on the fly, the boot order is computed anew with each boot, all the services on the same tree level are started at the same time and just have to fight it out. Thus the actual boot order varies greatly with each boot and will fail randomly, wait randomly (hands up: who wasn't annoyed by the systemd-rememberance-minute-and-a-half popping up like an inopportune windows update?) and will be hard to debug because you just cannot reproduce anything that lead to that. Your boot will be littered with 'wait-for-something' units, 'sleep-for-10s-because-that-magically-makes-it-work' and additional dependencies to maybe clean up the tree which the maintainers forgot about. Sometimes intentionally because "systemd is about making it boot fast, and it works on my laptop".
It isn't all bad, but for systems that I need to boot and shutdown reliably, systemd is a step back even over init scripts because of the nondeterminism. You can "fix" things by just watchdog-rebooting when the system doesn't come up, and the downtime will just be a little longer. But it is annoying. And it is a design bug that will never be fixed.
From my own experience the alternatives are fine as long as the distro maintainers put on the effort to make them work.
I've used Devuan and others with runit, sysV, s6 and openrc. And honestly, openrc is the better alternative from an user perspective.
Nevertheless, it is hard to argue against systemd when the critical mass is there, and you run into eventual problems.
Still, I find alternatives such as runit and openrc easier to reason about than systemd. Perhaps this is due to the need to fully grok them, so you're able to fix issues yourself. Contrast that with systemd, where there are plenty of drag and drop solutions out there, so that requirement is lessened. You could also argue the real reason is systemd's complexity, but personally that doesn't feel like the core reason.
Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
We made heavy use of systemd and its related components on a fleet of a few thousand edge devices and it worked great for us. Occasionally, we would run into bugs or confusing behaviour and have to dig through GitHub issues, but that was far outweighed by the problems it took care of for us.
systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
Not the person you're asking, but I'm curious what you mean when you say "the alternatives".
I've used runit, openrc, briefly upstart (what a trashfire that was), and I don't miss a single thing about them while writing systemd unit files.
I've used GNU Shepherd on Guix, and honestly, meh, systemd feels way more pragmatic.
I've also used systemd-networkd as well as NetworkManager, and feel like they both still have a few warts, but overall prefer systemd-networked on servers.
I've used grub, refit, and systemd-boot, and gotta say systemd-boot is my preference there too.
I haven't really used systemd portable services, but I've used docker and don't like it, and my overall impression is I'll probably like portable services more.
Are there other alternatives I have to use before I'm allowed to have an opinion? What alternatives have you used which made you feel so enlightened?
Blatantly not the person you're asking, but once you go past basic service management trying to do it just about anywhere else is an absolute and total nightmare. People who have major problems with systemd more than anything just ever seem confused, often vastly overestimating systemd's actual scope.
The reality to many is its the simplest simple way to handle some very complex tasks, and the vast majority of it is still all so very very much optional even in the most 'systemd heavy' distros.
Is ParticleOS also a useful everyday driver for Linux enthusiasts who aren't systemd developers? Let's say I got bored with Gentoo and I want to try something new. Is ParticleOS something that a technical user would enjoy at the current stage?
I think you'll get the most out of this at the moment if you're interested in actively contributing to systemd. There's lots of work left to be done to make this usable which will just be annoying if you aren't interested in fixing some of this stuff yourself.
Of course if you're interested in doing small prs and such to improve things then I think it could be very satisfying. You can always join the mkosi matrix channel to chat more about this stuff.
I'm not sure nix is aimed at the same kind of things here. ParticleOS seems to be a logical continuation of the effort that systemd has been putting into TPM2, boot security, signing, and immutability.
> Currently both arch and fedora are supported distributions.
Does this mean that we can now have an actual immutable Arch-based distro?!
I love Silverblue, best distro ever, but the day I can have immutable Arch Linux I’m outta here! I truly despise rpm and all the fedora machinations required to build a custom package.
Yeah I've been running the Arch variant of this on my personal laptop for a while now. Of course no stability guarantees with this for now since you'll be running systemd from source.
I work in the embedded space. I'd absolutely love to have a tool to build immutable, signed distro images which I can push to devices with a/b style updates. I imagine you can do this with mkosi, but it doesn't quite feel like the intended use case.
Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
Right now the industry standard tooling for building truly immutable images for embedded devices is Yocto. It works well, but it's incredibly complicated and has ridiculous build times because it insists on building every single package from source. It's utter madness that building a Linux image for a common ARM64 processor requires me to compile the Rust compiler myself.
You don't have to build ParticleOS images on the machine itself, it's perfectly possible to build them offline somewhere else and download them from the target machine when doing an update with systemd-sysupdate. It's just that we haven't quite gotten to ironing out all the details there. We're adding support to OBS (OpenSUSE Build System) to build ParticleOS images and will eventually start publishing prebuilt images there which can then be consumed by systemd-sysupdate.
For the embedded space you'd just build the ParticleOS images on your own build system, publish them somewhere and consume them with systemd-sysupdate, doesn't have to be OBS of course.
But we don't do stuff like only downloading diffs with systemd-sysupdate and such yet, so your milleage may vary.
That's interesting. I've been looking for an angle on embedded Linux software updates for my deployment tool (currently limited to updating desktop and server apps only).
When you say push, do you literally mean push or do you mean the devices would still pull updates on their own schedule where you get to control rollout percentages, channels, etc centrally? Mostly devices have to be assumed to be behind NAT, right?
What I'm thinking here is maybe it'd be useful to have a mode in Conveyor [1] that builds a standalone ARM Linux that can boot and pull updates for everything including the app, but which coordinates with the userspace app to decide when it's time to restart and for which the build to update to is controlled by server-supplied metadata so you can divide the fleet by tag, etc.
[1] https://hydraulic.dev/
The problem with immutable images is you need a complimentary config management tool to manage them at scale. That's why we built Etcha[0] (config management for 1000+ devices) and then built an immutable OS (EtchaOS) around it--they're meant to work together instead of "cloud-init is good enough". cloud-init is terrible at scale, especially outside of public cloud.
0 - https://etcha.dev
If only Tiny Core Linux had had better tooling and if only corporations liked simplicity, it might have been more popular…
Bulidroot is popular and simple alternative. But corporations prefer complexity.
2 replies →
> Most immutable distros (this included) are developed with the idea that you'd run local commands on the machine, and update a single OS image in place. That gets pretty unwieldy to manage across a fleet of devices.
This is how it always has been on enterprise network devices like switches, routers, etc. It’s pretty trivial to automate updates.
I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
>Sometimes ParticleOS adopts systemd features as soon as they get merged into systemd without waiting for an official release. As a result it's recommended to build systemd from source when building ParticleOS to make sure all required features are supported
> I’m somewhat a fan of systemd and I’m interested in immutable images, but I don’t want to build systemd from source:
You don't have to, we build packages and publish repositories from latest main, and the particle configs can use those by enabling the 'obs' build profile
> I’m somewhat a fan of systemd
As in: "you've tried alternatives and you really think that systemd is superior", or as in "you don't see why people complain about systemd and its philosophy"?
Genuinely interested, because usually people I hear say that they don't see the problem with systemd don't really have experience with the alternatives. Which makes sense: people who like being able to use alternatives tend to get screwed by systemd :-).
Edit: for those downvoting, this is a genuine question. I mostly hear opinions from people who hate systemd, and those who don't generally just don't really care: "I've always used systemd and I haven't had issues". But the latter group wouldn't call themselves "fans of systemd".
Systemd makes me sad because 99% of what I like about it could have been implemented in a modular fashion, but instead it's mostly all-or-nothing. I still use OpenRC on servers, but I have to implement or integrate a lot of the missing features, like all the cgroup/namespace stuff, logging, etc.
10 replies →
I've used Solaris SMF back in the day, which is what systemd largely copied, but badly.
The major difference is, in SMF you specify services, their dependencies and whatever else you need. Then SMF looks at the whole desired system, computes the dependency tree and boot order, validates everything and writes a boot plan that can be executed in a static and deterministic fashion. You can look at the plan, execute that exact same plan each boot. You therefore know that if the system came up once using that plan it will again.
Whereas systemd does everything on the fly, the boot order is computed anew with each boot, all the services on the same tree level are started at the same time and just have to fight it out. Thus the actual boot order varies greatly with each boot and will fail randomly, wait randomly (hands up: who wasn't annoyed by the systemd-rememberance-minute-and-a-half popping up like an inopportune windows update?) and will be hard to debug because you just cannot reproduce anything that lead to that. Your boot will be littered with 'wait-for-something' units, 'sleep-for-10s-because-that-magically-makes-it-work' and additional dependencies to maybe clean up the tree which the maintainers forgot about. Sometimes intentionally because "systemd is about making it boot fast, and it works on my laptop".
It isn't all bad, but for systems that I need to boot and shutdown reliably, systemd is a step back even over init scripts because of the nondeterminism. You can "fix" things by just watchdog-rebooting when the system doesn't come up, and the downtime will just be a little longer. But it is annoying. And it is a design bug that will never be fixed.
6 replies →
From my own experience the alternatives are fine as long as the distro maintainers put on the effort to make them work.
I've used Devuan and others with runit, sysV, s6 and openrc. And honestly, openrc is the better alternative from an user perspective.
Nevertheless, it is hard to argue against systemd when the critical mass is there, and you run into eventual problems.
Still, I find alternatives such as runit and openrc easier to reason about than systemd. Perhaps this is due to the need to fully grok them, so you're able to fix issues yourself. Contrast that with systemd, where there are plenty of drag and drop solutions out there, so that requirement is lessened. You could also argue the real reason is systemd's complexity, but personally that doesn't feel like the core reason.
Either way, it's good that there are alternatives. I like how heterogenous the whole Linux ecosystem can be, it helps mitigate large scale security issues.
4 replies →
We made heavy use of systemd and its related components on a fleet of a few thousand edge devices and it worked great for us. Occasionally, we would run into bugs or confusing behaviour and have to dig through GitHub issues, but that was far outweighed by the problems it took care of for us.
systemd does a lot more than just init. You can preach about whatever alternative inits all you like, but they don't do even 10% of what systemd does.
15 replies →
Not the person you're asking, but I'm curious what you mean when you say "the alternatives".
I've used runit, openrc, briefly upstart (what a trashfire that was), and I don't miss a single thing about them while writing systemd unit files.
I've used GNU Shepherd on Guix, and honestly, meh, systemd feels way more pragmatic.
I've also used systemd-networkd as well as NetworkManager, and feel like they both still have a few warts, but overall prefer systemd-networked on servers.
I've used grub, refit, and systemd-boot, and gotta say systemd-boot is my preference there too.
I haven't really used systemd portable services, but I've used docker and don't like it, and my overall impression is I'll probably like portable services more.
Are there other alternatives I have to use before I'm allowed to have an opinion? What alternatives have you used which made you feel so enlightened?
3 replies →
Blatantly not the person you're asking, but once you go past basic service management trying to do it just about anywhere else is an absolute and total nightmare. People who have major problems with systemd more than anything just ever seem confused, often vastly overestimating systemd's actual scope.
The reality to many is its the simplest simple way to handle some very complex tasks, and the vast majority of it is still all so very very much optional even in the most 'systemd heavy' distros.
Is ParticleOS also a useful everyday driver for Linux enthusiasts who aren't systemd developers? Let's say I got bored with Gentoo and I want to try something new. Is ParticleOS something that a technical user would enjoy at the current stage?
I think you'll get the most out of this at the moment if you're interested in actively contributing to systemd. There's lots of work left to be done to make this usable which will just be annoying if you aren't interested in fixing some of this stuff yourself.
Of course if you're interested in doing small prs and such to improve things then I think it could be very satisfying. You can always join the mkosi matrix channel to chat more about this stuff.
Nix would let you do just this already with potential binary cache and reproducibility.
I’m surprised it’s not more widely used for embedded. Maybe due to larger images? That’s a solvable problem.
Maybe best of all it’s fast. Yocto is painfully slow.
I'm not sure nix is aimed at the same kind of things here. ParticleOS seems to be a logical continuation of the effort that systemd has been putting into TPM2, boot security, signing, and immutability.
> Currently both arch and fedora are supported distributions.
Does this mean that we can now have an actual immutable Arch-based distro?!
I love Silverblue, best distro ever, but the day I can have immutable Arch Linux I’m outta here! I truly despise rpm and all the fedora machinations required to build a custom package.
Yeah I've been running the Arch variant of this on my personal laptop for a while now. Of course no stability guarantees with this for now since you'll be running systemd from source.
How does the update process work? Do you run pacman -Syu and then bake the result into the next image with mkiso?
1 reply →
How have your experiences with this setup been so far? Any major pain points?
2 replies →