Comment by chaps
3 hours ago
As someone who's lost many hours troubleshooting systemd failures, I would like an answer to this question, too.
3 hours ago
As someone who's lost many hours troubleshooting systemd failures, I would like an answer to this question, too.
You won't believe how many hours we have lost troubleshooting SysV init and Upstart issues. systemd is so much better in every way, reliable parallel init with dependencies, proper handling of double forking, much easier to secure services (systemd-analyze security), proper timer handling (yay, no more cron), proper temporary file/directory handling, centralized logs, etc.
It improves on about every level compared to what came before. And no, nothing is perfect and you sometimes have to troubleshoot it.
"In every way"
About ten years ago I took a three day cross-country Amtrak trip where I wanted to work on some data analysis that used mysql for its backend. It was a great venue for that sort of work because the lack of train-internet was wonderful to keep me focused. The data I was working with was about 20GB of parking ticket data. The data took a while to process over SQL which gave me the chance to check out the world unfolding outside of the train.
At some point, mysql (well, mariadb) got into a weird state after an unclean shutdown that put itself into recovery mode where upon startup it had to do some disk-intensive cleanup. Thing is -- systemd has a default setting (that's not readily documented, nor sufficiently described in its logs when the behavior happens) that halts the service startup after 30 seconds to try again. On loop.
My troubleshooting attempts were unsuccessful. And since I deleted the original csv files to save disk space, I wasn't able to even poke at the CSV files through python or whatnot.
So instead of doing the analysis I wanted to do on the train, I had to wait until I got to the end of the line to fix it. Sure enough, it was some default 30s timeout that's not explicitly mentioned nor commented out like many services do.
So, saying that things are "much better in every way" really falls on deaf ears and is reminiscent of the systemd devs' dismissive/arrogant behavior that many folk are frustrated about.
I had a situation like that with an undocumented behavior and systemd-tmpfiles. I wanted it to clean up a directory in /var/tmp/ occasionally. The automation using that directory kept breaking, however, because instead of either finding a whole intact git repo to update or a deleted repo, it instead found only a scattering of files that were root-owned with read-only permissions. There was yet another undocumented feature in systemd-tmpfiles where it would ignore root-owned, read-only files regardless of explicit configuration telling it to clean up the contents of those directories. Eventually this feature was quietly removed:
https://bugzilla.redhat.com/show_bug.cgi?id=1780979
https://github.com/systemd/systemd/commit/a083b4875e8dec5ce5...
That was far from the only time that the systemd developers decided to just break norms or do weird things because they felt like it, and then poorly communicate that change. Change itself is fine, it's how we progress. But part of that arrogance that you mentioned was always framing people who didn't like capricious or poorly communicated changes as being against progress, and that's always been the most annoying part of the whole thing.
> systemd is so much better in every way,
How can I cancel a systemd startup task that blocks the login prompt? / how is forcing me to wait for dhcp on a network interface that isn't even plugged in a better experience?
Your distribution has configured your GDM or Getty to have some dependency on something that ultimately waits on dhcpcd/network-online.target.
It’s not really the fault of systemd; it just enables new possibilities that were previously difficult/impossible and now the usage of said possibilities is surfacing problems.
1 reply →
There’s a reason why Devuan (a non systemd Debian) exists. Don’t want to get into a massive argument, but there are legitimate reasons for some to go in a different direction.
And Void Linux. And Gentoo. And Alpine Linux. And Slackware. And others.
4 replies →
Here are a few examples of problems systemd has caused me:
System shutdown/reboot is now unreliable. Sometimes it will be just as quick as it was before systemd arrived, but other times, systemd will decide that something isn't to its liking, and block shutdown for somewhere between 30 seconds and 10 minutes, waiting for something that will never happen. The thing in question might be different from one session to the next, and from one systemd version to the next; I can spend hours or days tracking down the process/mount/service in question and finding a workaround, only to have systemd hang on something else the next day. It offers no manual skip option, so unless I happen to be working on a host with systemd's timeouts reconfigured to reduce this problem, I'm stuck with either forcing a power-off or having my time wasted.
Something about systemd's meddling with cgroups broke the lxc control commands a few years back. To work around the problem, I have to replace every such command I use with something like `systemd-run --quiet --user --scope --property=Delegate=yes <command>`. That's a PITA that I'm unlikely to ever remember (or want to type) so I effectively cannot manage containers interactively without helper scripts any more. It's also a new systemd dependency, so those helper scripts now also need checks for cgroup version and systemd presence, and a different code path depending on the result. Making matters worse, that systemd-run command occasionally fails even when I do everything "right". What was once simple and easy is now complex and unreliable.
At some point, Lennart unilaterally decided that all machines accessed over a network must have a domain name. Subsequently, every machine running a distro that had migrated to systemd-resolved was suddenly unable to resolve its hostname-only peers on the LAN, despite the DNS server handling them just fine. Finding the problem, figuring out the cause, and reconfiguring around it wasn't the end of the world, but it did waste more of my time. Repeating that experience once or twice more when systemd behavior changed again and again eventually drove me to a policy of ripping out systemd-resolved entirely on any new installation. (Which, of course, takes more time.) I think this behavior may have been rolled back by now, but sadly, I'll never get my time back.
There are more examples, but I'm tired of re-living them and don't really want to write a book. I hope these few are enough to convey my point:
Systemd has been a net negative in my experience. It has made my life markedly worse, without bringing anything I needed. Based on conversations, comments, and bug reports I've seen over the years, I get the impression that many others have had a similar experience, but don't bother speaking up about it any more, because they're tired of being dismissed, ignored, or shouted down, just as I am.
I would welcome a reliable, minimal, non-invasive, dependency-based init. Systemd is not it.
I'd be interested in what other init alternatives offer the security options systemd does
The problem is not systemd vs SysV et al, the problem is systemd spreading like a cancer throughout the entire operating system.
Also trying to use systemd with podman is frustrating as hell. You just cannot run a system service using podman as a non-root user and have it work correctly.
Quadlet actually solves this. It's the newer way to define containers for systemd and handles the rootless user case properly. I migrated my services to it recently and it's much more robust than the old generate scripts.
5 replies →
> You just cannot run a system service using podman as a non-root user and have it work correctly.
Err... You just need to run `podman-compose systemd`?
I have my entire self-hosted stack running with systemd-controlled Podman, in regular user accounts.