Comment by andix
1 day ago
Slightly off topic: What's currently the free Linux distribution with the longest support cycle?
For a while I used CentOS 7 on all of those small VMs, because it got security updates for a really long time. With minimal risk of breaking things on updates.
PS: after a bit of research Alma/Rocky Linux are probably the best choices for now. 10 years of support. But are they maintained well?
For a while (a decade+), I was running CentOS on my servers on the same assumption of long time stability and ensuing peace of mind. Then I figured that over such durations, the ecosystem drift becomes significant and keeping applications up to date and running on top of the OS becomes an increasing challenge (with the more "infrastructure" packages like glibc, python/Apache combos, GCC, ... slowly becoming incompatible with the latest applicative stack).
Then I figured that version upgrades were miserable, not just because I had painted myself in a weird corner with ungodly packages mix-ups, but because the upgrade path was always best-effort. I think I gave up during the 6 to 7 transition, as I realised that all I needed was fedora: with yearly or half-yearly updates I have no need to fight the distro's packages: stuff stays current and in working order, major distro upgrades go smoothly, downtime is minimal. I'm not considering going back to any "server distribution" ever.
I see no reason not to go with a rolling release distro for personal servers. Run all the services in containers and have the base OS auto-update itself as often as it needs.
Went with openSUSE MicroOS myself, it updates and reboots almost daily so I can be pretty confident my server is healthy and it's atomic so if something does break and I don't feel like dealing with it, I can just click rollback button from cockpit and deal with it whenever I have time.
Everything you listed is the antithesis to managing and maintaining high availability systems
It's your personal toy server, you are optimizing for something entirely different than high availability.
1 reply →
Then you also have to auto-update the containers, if it's a public facing service. Either you'll have to build containers yourself or hope the developer pushes a new update whenever the base image has relevant security fixes.
Yup, podman quadlets autoupdate quite nicely. Setting up a local registry mirror with ~3d delay before applying updates is on my todo list.
My own service images already have a script that runs daily that pulls latest git updates and builds fresh images.
>!!!I see no reason not to go with a rolling release distro!!! for personal servers. Run all the services in containers and !!!have the base OS auto-update itself as often as it needs.!!!
you do not belong in IT
Personal servers.
There are things that need 9^5 and there are things that don't. If someone backs up their application configs and data properly, then the only thing that really matters is a proper backup strategy.
All my critical files are backed up periodically (manually) via rclone to S3 glacier, and all my services are documented in dokuwiki. If you use ansible or want to store configs and installation scripts, a private git repo would do well.
After that, I don't see a problem running rolling or short-support OS like Fedora Server for application hosting.
1 reply →
> But are they maintained well?
Alma has a few affordances as it's no longer RHEL source compatible, which means it could ship priviledge escalation fixes with new kernel updates faster.
Rocky responded with an extra, optional to enable, security repo to provide mitigations to the exploits while waiting for RHEL to downstream.
Look pretty well maintained to me. If only judging by recent events.
Rocky's docs are also really nice. They aren't as thorough as RedHat's, but they're much more readable and concise, and tend to be written for a less enterprise-y audience.
Don't even remind me about the RedHat docs, lol. Their solutions pages used to be readable with an account, now I think you need a subscription too.
The manuals, indeed are good, though for more esoteric issues I land too often on a gated answer page.
2 replies →
Thanks!
I don't care much about being fully RHEL compatible, or no ABI changes at all. I just want a system that gets security fixes quickly with as little chances of breaking things as possible.
How about a lightweight immutable distro, like say Fedora CoreOS or openSUSE MicroOS?
Fedora CoreOS in particular has had a good track record delivering patches quickly. Like for CopyFail was pushed to the stable channel in about a day, IIRC, but the patch was already available within a few hours of disclosure in the "next" / testing channel.
Talos and Flatcar are also worth considering if you want an even smaller attack surface, from what I heard they weren't even affected by CopyFail.
3 replies →
You are betting that whatever you host doesn't live as long as the upgrade cycle because it'll probably be a pain when the upgrades finally arrive. I'd rather have smaller version jumps more often than a huge jump with everything changing after a long time.
It usually doesn't live until the end of the support cycle. And if it does I will probably migrate it to a fresh VM instead of upgrading the distribution.
I'm not worried about upgrading. I'm worried about the whole environment being potentially several versions newer than the old one. All shared libraries. All services. Everything new. And now you have to make a software that has had little upgrades run on that. Have fun.
Completely agree. A fresh install beats an in-place major version upgrade every time. Less hazardous and gives an easy path to clear out all the accumulated crud.
Alma and Rocky if you want fully free or have a lot of machines. RHEL if you are okay with registering with them; they give ten machines free access to their updates for each Registered account in their system.
RHEL is definitely the most stable major distribution. Alma and Rocky are essentially downstream clones of RHEL.
I would say NixOS, where it is trivial to switch across releases, run software from different releases, and perform rollbacks.
I have been running NixOS on several servers for more than a decade. No reinstalling, upgrading, or any breaks whatsoever.
This is your personal opinion, a rolling release like NicOS is exactly the opposite of an LTS distro.
I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden.
> a rolling release like NixOS is exactly the opposite of an LTS distro
NixOS is not rolling release. This is a common misconception. You can use the unstable channel, which is a rolling release, or the regular channels which get released twice a year. These are really stable and move very slowly. You can also mix and match, running software from different channels.
> I actually wonder what would happen to a NixOS installation frozen in time for 5 years that then you want to update to latest all of a sudden
I have done this recently as I kept an airgapped machine, which I decommissioned, connected to the Internet and updated to the latest channel. Everything worked just fine. I just had to change a couple of options in my configuration which had become outdated. Nix is functional, so it's much less prone to all stateful issues that plague other package managers.
I'd say not much: you update the channel, run nixos-rebuild switch, fix all the warnings/errors due to renamed/changed options until it succeeds and you're done. If you have a database like postgres you may have to do a schema upgrade manually, since the default version is updated every 4/5 releases or so.
It's very rare to find something that prevents you from directly updating. Nixpkgs tries very hard to no require new Nix features, so it evaluates with even Nix versions from a decade ago. Also, NixOS options and packages are frequently changed, but the automatic migrations (mkChangedOptionModule, mkRenamedOptionModule, alias, etc.) are never removed in practice.
Since the binary cache has never been cleared since its creation (2002?), it should actually be easy to install a super old NixOS release and upgrading it to the latest to see what happens.
By the way, there are LTS versions of NixOS, just not officially supported. See https://docs.ctrl-os.com/.
I've only been running NixOS (in any serious capacity) for three years, but I have installed it on every computer that I am allowed to install it on now.
It has been the most headache-free Linux I've used, simply because I'm less scared to play with and fix stuff. The fact that rollbacks are trivial and snapshots are automatic, and since everything is declarative in a text file anyway, I am way braver. If I do something like screw up the video driver, or the wifi driver or make it so the system doesn't boot anymore, all I need to do is reboot and choose a previous generation.
> simply because I'm less scared to play with and fix stuff.
The main reason of a LTS distribution is not having to play around and fix stuff. Install something once, and it keeps running without any changes, but still gets security updates.
1 reply →
Earlier today, I tried to run a simple nix tool a colleague made. 3 hours into the build, it crashed. Something about a missing python import? I ran the exact same ‘nix develop’ again. 2 hours later, it worked.
Keep in mind: this was just a simple rest server. But for some reason it needed to (nondeterministically) build the word from scratch to send that single request.
I’ll take a docker system thank you.
I have run NixOS for about eight years on server and desktop and been a nixpkgs maintainer. Yes, most of the time I would agree with you. The fact that you get warnings in the terminal for a lot of incompatibilities and changes when upgrading is a really nice touch and upgrades tend to be smooth. I do not use rollbacks much, but when you do need them they are really handy. Having every configuration in a single file makes you more bold to play around with configurations, which felt really empowering when I first got into NixOS, as I knew it could roll things back and I no longer had to keep notes on how each box was set up to refer to in the case of a reinstall or migration.
However, I have had one machine become unbootable as it could no longer mount its encrypted disks after an upgrade, forcing me to mount a rescue image remotely, mount the disks manually, lift the data out, and do a complete reinstall (migrated the box to OpenBSD at that time). Similarly, NixOS once messed up systemd (or vice versa) so badly that I could not even reboot without forcing a power cycle. Lastly, I have had a package break for my use cases by maintainers enabling so many custom flags by default for a package that they enabled one I have never seen enabled by any other packaging team and that then broke RTSP in "funny" ways. Ubuntu did tend to break things like graphics between releases at times back when I used it, but I have never had any other distribution or operating system throw curve balls like the three things I mentioned here.
My general impression of NixOS is that the core is solid, but that nixpkgs just has such a large number of things that it supports that the maintainers struggle to test them all and can not anticipate the interactions between all the packages and options. The default Julia package being so broken that it produced incorrect mathematics due to nixpkgs' insistence on allowing you to swap out the Blas library and also having turned off the unit tests for example springs to mind. This was shipped to end users for a long time before I noticed it by accident by enabling the unit tests and stepped in to clean it up. It all feels very "Gentoo", which was indeed an inspiration for NixOS by the way.
Now, return to that last sentence in the first paragraph that I wrote about feeling empowered to tinker, ultimately, I feel like you should try to resist that urge as it is what pushes you into the untested fractal of possible configurations that NixOS allows you to explore. My other main operating system is OpenBSD, where the mentality is "Stick to the defaults or suffer the consequences"; with NixOS, I feel like everyone's box is more or less a tailored suit, which comes with both its ups and downs.
I run nixOS as well on my home infrastructure (gateway/firewall, a couple of internal servers).
But I have had, uh, non-trivial breakages happen also when I upgrade the system itself to the next yearly release. Non-bootable kernel kind of breakages.
But I will give you that I can just boot from the generation before the upgrade, and it works again. So there's that :)
Eh nix flakes are a nightmare to configure. Far more verbose than a docker compose. They rely on some caches which keep pre-compiled packages and you better make sure you have the caches with the particular flakes you need set up. Yikes
I don't have data, but my guess would be Debian or Slackware
No
Ubuntu Pro is free for up to 5 systems. 15 years.
Debian LTS/extended LTS
5 years is not a lot. It releases every 2 years, so it requires upgrading at least every 4 years. In the worst case it's just 3 years of support, if you install right before the next release.
ELTS is 10 years and paid. It's great that it exists, but not relevant for my toy projects.
I feel there is a balance to be struck between a project that is popular (where if you run into problems, you will get good support), and one that technically gives longer-term support (but if things go wrong, that support might not be very good).
I haven't used a lot of different distros, but for me, Debian has been a good balance of those factors. You may need to do more upgrades per decade, but the ones that you do are more liable to go smoothly.
Just my 2¢ on the topic (:
1 reply →
I don't work on a server team, but in network/network security. My company made an announcement that they are extending our product's software lifetime to four years: 3 years standard support + 1 year high sev patches.
It seems to me in the 2020s that 5-7 years is plenty of support for a single OS release, and that OS support teams should be nimble enough to roll out new instances and migrate data at that cadence.
1 reply →
So there is a project that you care enough about to keep it alive, but 1-2 hours every FOUR YEARS is too much? At some point I just have to call you lazy dude.
Either the 1-2 hours is a drop in the bucket compared to what you spend on it anyway (like a blog you still regularly update), or you don't actively update the project but still care enough about it to spend half an evening every few years, or you should just admit you don't care about it enough anymore to do even that. In the last case just delete the project.
4 replies →
Probably Debian or Ubuntu. The question is...why do you care that much?
I've upgraded Debian stable (both pure and with some cherry-picked backports) and Ubuntu (non-LTS and LTS) systems in place and rarely broken anything, for years and years. When stuff has broken it's been a quick google and then slapping myself for not having read the upgrade guide.
I do generally wait about 2-3 weeks before upgrading, giving time for them to catch stuff that was missed until the great masses were set loose on it.
> The question is...why do you care that much?
Not the OP, but I support Ubuntu as desktop and server OS for an engineering collage and have for 10ish years. Some LTS upgrades don't require many changes (mostly minor package name changes) and some take months of work to get rolled out (mostly for workstations, the server upgrades are usually quick.). Not everything gets upgraded every new OS release. If we had to upgrade everything every 6-12 months it would eat up a significant amount of time for our small team.
Only using ubuntu rn, but when the server is mostly running docker, it is simpler upgrade nowadays with so little dependencies. But then the problem just moved to the container image updates.
I have a machine that has been Fedora since twenty-something to current 44, and upgrading yearly is a breeze. Three commands, and just wait for a download and the reboot. The only thing that breaks if you forget that the upgrade needs attention is the system Postgres, until I migrated to Podman images.
1 reply →
> why do you care that much?
I've had issues with Ubuntu/Debian upgrades more than once. Some third party binaries breaking with the update. Or some specific config tweaks that break, because the structure of /etc changed too much.
For some small VM with a specific purpose I prefer a distribution that changes as little as possible for as long as possible. Less work, more uptime.
I won't touch ubuntu unless forced to by some obscure work requirement. I've had enough bad experiences with repos being shut down, updates/upgrades breaking unanticipated, obscure things, and I hate snap.
The naming conventions drive me crazy as well. When you deal with 2 things that have dumbshit naming conventions, like ubuntu and ROS, its really obnoxious to pretend to case enough to keep track of.
Ive had nothing but issues doing that. I think I’ve had a Debian upgrade actually succeed maybe one time? (After some manual intervention to fix some issue other booting on my work server)
For updates, Debian and Ubuntu are great. For upgrades… not so much for me.
I had unattended-upgrades cripple our VMs
I mean Ubuntu Pro is free for personal use and it extends the LTS support of 5 years so a total of 10 years afaik.
[dead]
Use a rolling release like Arch and it’s supported forever.
But then you have constant maintenance. I prefer rolling distros, don't get me wrong. But it does mean you will get the latest of every package constantly and some cause problems.
For a box that sits in a corner doing its joband you don't want to pay attention to it's not a good choice IMO. On a desktop you want the latest of everything on and you have time to keep up it's the best.
I have an Arch server that has been online for ten years (yikes), never had any issues with it.
1 reply →
I need to enable automatic updates, because I don't have the time to manually update. I have a few machines on Open SuSE Tubleweed, and stuff just randomly breaks. A few months ago there was a weird Kernel bug that just froze all of them. They update and reboot every day, and suddenly it all worked well again. A bit too exciting for me :)
You can always try openSUSE Slowroll (in beta), which is a rolling release that updates less frequently than Tumbleweed. It advertises better stability.
https://en.opensuse.org/Portal:Slowroll