Comment by senfiaj
1 day ago
This might offend some people but even Linus Torvalds thinks that the ABI compatibility is not good enough in Linux distros, and this is one of the main reasons Linux is not popular on the desktop. https://www.youtube.com/watch?v=5PmHRSeA2c8&t=283s
To quote a friend; "Glibc is a waste of a perfectly good stable kernel ABI"
Kind of funny to realize, the NT kernel ABI isn’t even all that stable itself; it is just wrapped in a set of very stable userland exposures (Win32, UWP, etc.), and it’s those exposures that Windows executables are relying on. A theoretical Windows PE binary that was 100% statically linked (and so directly contained NT syscalls) wouldn’t be at-all portable between different Windows versions.
Linux with glibc is the complete opposite; there really does exist old Linux software that static-links in everything down to libc, just interacting with the kernel through syscalls—and it does (almost always) still work to run such software on a modern Linux, even when the software is 10-20 years old.
I guess this is why Linux containers are such a thing: you’re taking a dynamically-linked Linux binary and pinning it to a particular entire userland, such that when you run the old software, it calls into the old glibc. Containers work, because they ultimately ground out in the same set of stable kernel ABI calls.
(Which, now that I think of it, makes me wonder how exactly Windows containers work. I’m guessing each one brings its own NTOSKRNL, that gets spun up under HyperV if the host kernel ABI doesn’t match the guest?)
IIRC, Windows containers require that the container be built with a base image that matches the host for it to work at all (like, the exact build of Windows has to match). Guessing that’s how they get a ‘stable ABI’.
…actually, looks like it’s a bit looser these days. Version matrix incoming: https://learn.microsoft.com/en-us/virtualization/windowscont...
1 reply →
> Kind of funny to realize, the NT kernel ABI isn’t even all that stable itself
This is not a big problem if it's hard/unlikely enough to write a code that accidentally relies on raw syscalls. At least MS's dev tooling doesn't provide an easy way to bypass the standard DLLs.
> makes me wonder how exactly Windows containers work
I guess containers do the syscalls through the standard Windows DLLs like any regular userspace application. If it's a Linux container on Windows, probably the WSL syscalls, which I guess, are stable.
> NT kernel ABI isn’t even all that stable itself
Can you give an example where a breaking change was introduced in NT kernel ABI?
15 replies →
Apparently there are 3 kinds of Windows containers, one using HyperV, and the others sharing the kernel (like Linux containers)
https://thomasvanlaere.com/posts/2021/06/exploring-windows-c...
Docker on windows isn't simply a glorified virtual machine running a Linux. aka Linux subsystem v2
At least glibc uses versioned symbols. Hundreds of other widely-used open source libraries don't.
Versioned glibc symbols are part of the reason that binaries aren't portable across Linux distributions and time.
17 replies →
Yeah and nothing ever lets you pick which versions to link to. You're going to get the latest ones and you better enjoy that. I found it out the hard way recently when I just wanted to do a perfectly normal thing of distributing precompiled binaries for my project. Ended up using whatever "Amazon Linux" is because it uses an old enough glibc but has a new enough gcc.
3 replies →
> Hundreds of other widely-used open source libraries don't.
Correct me if I'm wrong but I don't think versioned symbols are a thing on Windows (i.e. they are non-portable). This is not a problem for glibc but it is very much a problem for a lot of open source libraries (which instead tend to just provide a stable C ABI if they care).
6 replies →
I only learned about glibc earlier today, when I was trying to figure out why the Nix version of a game crashes on SteamOS unless you unset some environ vars.
Turns out that Nix is built against a different version of glibc than SteamOS, and for some reason, that matters. You have to make sure none of Steam's libraries are on the path before the Nix code will run. It seems impractical to expect every piece of software on your computer to be built against a specific version of a specific library, but I guess that's Linux for you.
Ask your friend if he would CC0 the quote or similar (not sure if its possible but like) I can imagine this being a quote on t-shirts xD
Honestly I might buy a T-shirt with such a quote.
I think glibc is such a pain that it is the reason why we have so vastly different package management and I feel like non glibc things really would simplify the package management approach to linux which although feels solved, there are definitely still issues with the approach and I think we should still all definitely as such look for ways to solve the problem
Non-glibc distros (musl, uclibc...) with package managers have been a thing for ages already.
1 reply →
AppImage, theoretically, solves this problem (or FlatPak I guess). The issue would really be in getting people to package up dead/abandoned software.
https://zapps.app/ is another interesting thing in the space.
AppImage have some issues/restrictions like it cant run on older linux than one it was compiled on, so people compile it on the oldest pc's and a little bit of more quirks
AppImage are really good but zapps are good too, I had once tried to do something on top of zapp but shame that zapp went into the route of crypto ipfs or smth and then I don't really see any development of that now but it would be interesting if someone can add the features of zapp perhaps into appimage or pick up the project and build something similar perhaps.
This is really cool. Looks like it has a way for me to use my own dynamic linker and glibc version *.
At some point I've got to try this. I think it would be nice to have some tools to turn an existing programs into a zapps (there many such tools for making AppImages today).
* https://github.com/warptools/ldshim
4 replies →
Interesting. I've had a hell of a time building AppImages for my apps that work on Fedora 43. I've found bug reports of people with similar challenges, but it's bizarre because I use plenty of AppImages on F43 that work fine. I wonder if this might be a clue
I can only speak for Flatpak, but I found its packaging workflow and restricted runtime terrible to work with. Lots of undocumented/hard to find behaviour and very painful to integrate with existing package managers (e.g. vcpkg).
Yeah, flatpak has some good ideas, and they're even mostly well executed, but once you start trying to build your own flatpaks or look under the hood there's a lot of "magic". (Examples: Where do runtimes come from? I couldn't find any docs other than a note that says to not worry about it because you should never ever try to make your own, and I couldn't even figure out the git repos that appear to create the official ones. How do you build software? Well, mostly you plug it into the existing buildsystems and hope that works, though I mostly resorted to `buildsystem: simple` and doing it by hand.) For bonus points, I'm pretty sure 1. flatpaks are actually pretty conceptually simple; the whole base is in /usr and the whole app is in /app and that's it, and 2. the whole thing could have been a thin wrapper over docker/podman like x11docker taken in a slightly different direction.
3 replies →
Appimage maybe, don’t say flatpak Cz wherever I update my arch system flatpak gets broken which I have to fix by updating or reinstalling
While true in many respects (still), it's worth pointing out that this take is 12 years old.
Maybe it's better now in some distros. Not sure about other distros, but I don't like Ubuntu's Snap package. Snap packages typically start slower, use more RAM, require sudo privileges to install, and run in an isolated environment only on systems with AppArmour. Snap also tends to slow things some at boot and shutdown. People report issues like theming mismatches, permissions/file-access friction. Firefox theming complaints are a common example. It's almost like running a docker container for each application. Flatpaks seem slightly better, but still a bandaid. Just nobody is going to fix the compatibility problems in Linux.
Ubuntu was getting too good so it had to snap half of its value out of existence.
You can still get firefox as a .deb though.
https://launchpad.net/~mozillateam/+archive/ubuntu/ppa
I think he still considers this to be the case. He was interviewed on Linus tech tips recently. And he bemoaned in passing the terrible application ecosystem on Linux.
It makes sense. Every distribution wants to be in charge of what set of libraries are available on their platform. And they all have their own way to manage software. Developing applications on Linux that can be widely used across distributions is way more complex than it needs to be. I can just ship a binary for windows and macOS. For Linux, you need an rpm and a dpkg and so on.
I use davinci resolve on Linux. The resolve developers only officially support Rocky Linux because anything else is too hard. I use it in Linux mint anyway. The application has no title bar and recording audio doesn’t work properly. Bleh.
I agree 100% with Linus. I can run a WinXP exe on Win10 or 11 almost every time, but on Linux I often have to chase down versions that still work with the latest Mint or Ubuntu distros. Stuff that worked before just breaks, especially if the app isn’t in the repo.
Yes and even the package format thing is a hell of its own. Even on Ubuntu you have multiple package formats and sometimes there are even multiple app stores (a Gnome one and an Ubuntu specific if I remember correctly)
Ultimately this boils down to lack of clear technical and community leadership from Canonical. Too unwilling to say "no" to vanity/pet projects that end up costing all of us as they make the resulting distribution into a moving target too difficult to support in the enterprise - at least, not with the skillset of the average desktop support hire these days.
I want to go to the alternate timeline where they just stuck with a set of technologies... ideally KDE... and just matured them up until they were the idealized version of their original plan instead of always throwing things away to rewrite them for ideological or technical purity of design.
You can also run a WinXP exe on any Linux distribution almost every time. That's the point of project and Linus' quip: The only stable ABI around on MS Windows and Linux is Win32 (BTW, I do not agree with this.)
I think it's not unlikely that we reach reach a point in a couple of decades where we are all developing win32 apps that most people are running some form of linux.
We already have an entire platform like that (steam deck), and it's the best linux development experience around in my opinion.
That’s actually an intentional nudge to make the software packaged by the distro, which usually implies that they are open source.
Who needs ABI compatibility when your software is OSS? You only need API compatibility at that point.
So every Linux distribution should compile and distribute packages for every single piece of open source software in existence, both the very newest stuff that was only released last week, and also everything from 30+ years ago, no matter how obscure.
Because almost certainly someone out there will want to use it. And they should be able to, because that is the entire point of free software: user freedom.
25 replies →
Even open-source software has to deal with the moving target that is ABI and API compatibility on Linux. OpenSSL’s API versioning is a nightmare, for example, and it’s the most critical piece of software to dynamically link (and almost everything needs a crypto/SSL library).
Stable ABIs for certain critical pieces of independently-updatable software (libc, OpenSSL, etc.) is not even that big of a lift or a hard tradeoff. I’ve never run into any issues with macOS’s libc because it doesn’t version the symbol for fopen like glibc does. It just requires commitment and forethought.
The reason you're getting downvoted is that what you're saying implies a shit-ton of work for the distros -- that's expensive work that someone has to pay for (but nobody wants to, and think of the cost of opportunity).
But you're not entirely wrong -- as long as you have API compatibility then it's just a rebuild, right? Well, no, because something always breaks and requires attention. The fact is that in the world of open source the devs/maintainers can't be as disciplined about API compat as you want them to be, and sometimes they have to break backwards compatibility for reasons (security, or just too much tech debt and maint load for obsolete APIs). Because every upstream evolves at a different rate, keeping a distro updated is just hard.
I'm not saying that statically linking things and continuing to run the binaries for decades is a good answer though. I'm merely explaining why I think your comment got downvoted.
Everyone is mentioning ABI, but this is really an API problem, so "you only need API compatibility at that point" is a very big understatement.
This might be why OpenBSD looks attractive to some. Its kernel and all the different applications are fully integrated with each other -- no distros! It also tries to be simple, I believe, which makes it more secure and overall less buggy.
To be honest, I think OSes are boring, and should have been that way since maybe 1995. The basic notions:
haven't changed since 1970, and the more modern GUI stuff hasn't changed since at least the early '90s. Some design elements, like
are perhaps even arbitrary, but can serve as a mature foundation for better-concieved ideas, such as:
I'm wondering why OSS - which according to one of its acolytes, makes all bugs shallow - couldn't make its flagship OS more stable and boring. It's produced an
> tree-like file systems, multiple users, access privileges,
Why should everything pretend to be a 1970s minicomputer shared by multiple users connected via teletypes?
If there's one good idea in Unix-like systems that should be preserved, IMHO it's independent processes, possibly written in different languages, communicating with each other through file handles. These processes should be isolated from each other, and from access to arbitrary files and devices. But there should be a single privileged process, the "shell" (whether command line, TUI, or GUI), that is responsible for coordinating it all, by launching and passing handles to files/pipes to any other process, under control of the user.
Could be done by typing file names, or selecting from a drop-down list, or by drag-and-drop. Other program arguments should be defined in some standard format so that e.g. a text based shell could auto-complete them like in VMS, and a graphical one could build a dialog box from the definition.
I don't want to fiddle with permissions or user accounts, ever. It's my computer, and it should do what I tell it to, whether that's opening a text document in my home directory, or writing a disk image to the USB stick I just plugged in. Or even passing full control of some device to a VM running another operating system that has the appropriate drivers installed.
But it should all be controlled by the user. Normal programs of course shouldn't be able to open "/dev/sdb", but neither should they be able to open "/home/foo/bar.txt". Outside of the program's own private directory, the only way to access anything should be via handles passed from the launching process, or some other standard protocol.
And get rid of "everything is text". For a computer, parsing text is like for a human to read a book over the phone, with an illiterate person on the other end who can only describe the shape of each letter one by one. Every system-level language should support structs, and those are like telepathy in comparison. But no, that's scaaaary, hackers will overflow your buffers to turn your computer into a bomb and blow you to kingdom come! Yeah, not like there's ever been any vulnerability in text parsers, right? Making sure every special shell character is properly escaped is so easy! Sed and awk are the ideal way to manipulate structured data!
Indeed.
AmigaOS was the pinnacle of personal computing OS design. Everything since has been a regression. Fite me.
4 replies →
I like FreeBSD for the same reason. The whole system is sane and coherent. Illumos is the same.
I wish either of those systems had the same hardware & software support. I’d swap my desktop over in a heartbeat if I could.
OpenBSD—all the BSDs really—have an even more unstable ABI than Linux. The syscall interface, in particular, is subject to change at any time. Statically linked binaries for one Linux version will generally Just Work with any subsequent version; this is not the case for BSD!
There's a lot to like about BSD, and many reasons to prefer OpenBSD to Linux, but ABI backward-compatibility is not one of them!
One of Linux's main problems is that it's difficult to supply and link versions of library dependencies local to a program. Janky workarounds such as containerization, AppImage, etc. have been developed to combat this. But in the Windows world, applications literally ship, and link against, the libc they were built with (msvcrt, now ucrt I guess).
Because Linux not an OS. The flagship OSS OS is Ubuntu, and it's mostly pretty stable. But OSS inherently implies the ability to make your own OS that's different from someone else's OS, so a bunch of people did just that.
Ubuntu still suffers the same kind of breakage though. You can't take an moderately complex GUI application that was built on ubuntu 2014 and run it on the latest version. Heck, there's a good chance you can't even build it on the newer version without needing to update it somehow. It's a property of the library ecosystem around linux, not the behaviour of a given distro.
(OK, I have some experience with vendors where their latest month-old release has an distro support release where the most up-to-date option is still 6 months past EOL, and I have managed to hack something together which will get them to work on the newer release, but it's extremely painful and very much not what either the distros or the software vendors want to support)
Isn't the kernel responsible for the ABI?
ABI is a far larger concept than the kernel UAPI. Remember that the OS includes a lot of things in userspace as well. Many of these things are not even stable between the various contemporary Linux distros, let alone older versions of them. This might include dbus services, fs layout, window manager integration, and all sorts of other things.
Yeah, it is almost like that complete OS should be called something else than "Linux".
Thanks
Android makes a sport of breaking ABI compatibly and it hasn't stopped it from being the most popular mobile OS
The reason being JetPack libraries that abstract what Android version is being used.
That's outright not true though.
What are you even talking about? Android is by far the most popular mobile OS worldwide. It's only in the US where iPhones are dominant.
Nvm I can't read
What's interesting to think about is Conway's law and monorepos and the Linux kernel and userland. If it were all just one big repo, then making breaking changes, wouldn't. The whole ifconfig > ip debacle is an example of where one giant monorepo would have changed how things happened.
It's really just glibc
It's really just not. GTK is on its fourth major version. Wayland broke backwards compatibility with tons of apps.
Multiple versions of GTK or QT can coexist on the same system. GTK2 is still packaged on most distros, I think for example GIMP only switched to GTK3 last year or so.
GTK update schedule is very slow, and you can run multiple major versions of GTK on the same computer, it's not the right argument. When people says GTK backwards compatibility is bad, they are referring in particular to its breaking changes between minor versions. It was common for themes and apps to break (or work differently) between minor versions of GTK+ 3, as deprecations were sometimes accompanied with the breaking of the deprecated code. (anyway, before Wayland support became important people stuck to GTK+ 2 which was simple, stable, and still supported at the time; and everyone had it installed on their computer alongside GTK+ 3).
Breaking between major versions is annoying (2 to 3, 3 to 4), but for the most part it's renaming work and some slight API modifications, reminiscent of the Python 2 to 3 switch, and it only happened twice since 2000.
The difference is that you can statically link GTK+, and it'll work. You can't statically link glibc, if you want to be able to resolve hostnames or users, because of NSS modules.
2 replies →
Can't we just freeze glibc, at least from an API version perspective?
We definitely can, because almost every other POSIX libc doesn’t have symbol versioning (or MSVC-style multi-version support). It’s not like the behavior of “open” changes radically all the time, and you need to know exactly what source symbol it linked against. It’s really just an artifact of decisions from decades ago, and the cure is way worse than the disease.
The problem is not the APIs, it's symbol versions. You will routinely get loader errors when running software compiled against a newer glibc than what a system provides, even if the caller does not use any "new" APIs.
glibc-based toolchains are ultimately missing a GLIBC_MIN_DEPLOYMENT_TARGET definition that gets passed to the linker so it knows which minimum version of glibc your software supports, similar to how Apple's toolchain lets you target older MacOS from a newer toolchain.
7 replies →
Or just pre-install all the versions on each distro and pick the right one at load-time