Comment by mikepavone
2 days ago
So of the 3 glibc issues they link
- one is about the format of symbol information in the actual ELF binaries which is only an issue if you are not using the standard libc functions for looking up symbols for some strange reason
- one is an issue that impacts targeting a lower version of glibc from a higher one which is a configuration that was never supported (though usually fails more loudly)
- the last one is a security policy change which is legitimately an ABI break, but mostly impacts programs that have their execstack flags set incorrectly
glibc actually goes to a fair bit of effort to be compatible with old binaries unlike most of the rest of the Linux userspace. The binaries I built for my side project back in 2015 (BlastEm 0.3.0) still work fine on modern Linux and they dynamically link against glibc. This is just a hobby project, not a piece of professional software, and a build from before this JangaFX company even existed works fine.
I find it really bizarre when people talk about Linux binary compat and then complain entirely about glibc rather than the sort of problems that the manylinux project has had to deal with. glibc is one of the few parts of userspace you can depend on. Yes, setting up your toolchain to build against an old glibc on a modern distro is a bit annoying. Sure, if you do something sufficiently weird you might find yourself outside what glibc considers part of their stable ABI. But from where I sit, it works pretty well.
I think the problem in a nutshell is that it's not trivial(?) to build an executable on a modern Linux distro that links against an old glibc version number (and if it is trivial then it needs to be better communicated).
It is actually quite trivial when building with the Zig toolchain since you can simply append the requested glibc version to the target-triple (e.g. `-target aarch64-linux-gnu.2.xx`), but I think this doesn't work with regular clang or gcc (makes one wonder why not when Zig can pull it off).
Just to add some more context. Zig cc is a wrapper around clang. It can handle cross compiling to specific glibc versions. See https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace... I imagine it would help with the glibc problems they are taking about. Glibc tries to provide a backwards compatible abi.
> I think the problem in a nutshell is that it's not trivial(?) to build an executable on a modern Linux distro that links against an old glibc version number (and if it is trivial then it needs to be better communicated).
I wouldn't say it's trivial, but it's not rocket science either. Basically there are two main approaches. One is to just build inside a chroot or container with a sufficiently old distro inside. This is generally the path of least resistance because your build system doesn't really have to have any awareness of what's going on. You just build normally inside the chroot/container. The main downsides with this approach are that it's kind of wasteful (you have a whole distro's filesystem) and if you want to use a newer compiler than what the old distro in question shipped with you generally have to build it yourself inside said chroot/container.
The other main approach is to use a sysroot. gcc and clang both take an optional --sysroot parameter which is an alternative root for header and library lookups. This lets you use a compiler on the normal host, but old headers and libs. You can also bake this parameter in when compiling gcc (and also I assume clang, but less sure there) if you want a dedicated cross-toolchain.
Is there a good summary of the problems manylinux had to deal with?
The PEP-600 [0] Rationale section touches on this a bit. The basic problem is that there are things beyond glibc that would be nice to use from the environment for a number of reasons (security updates, avoiding clashes between multiple wheels that depend on the same lib, etc.), but since most libs outside of glibc and libstdc++ don't really have an ABI policy and the distros don't necessarily have a policy on what libraries are guaranteed to be present you sort of have to guess and hope for the best. While the initial list in PEP-513 [1] was a pretty good guess, one of the libraries chosen (libcrypt.so.1) got dropped in Fedora 30 and replaced with an ABI incompatible version. Crypto libraries are an example of something that's actually important to keep up to date so I find this rather unfortunate.
[0] https://peps.python.org/pep-0600/
[1] https://peps.python.org/pep-0513/
>then complain entirely about glibc
You can ship all of your of the libraries you use with your executable. This isn't possible to do with glibc. It's the exception which is why it's talked about the most.
> It's the exception which is why it's talked about the most.
It's definitely not the only exception. libgl is another obvious example since different GPUs need different userland code. I would be surprised if there there had never been compat regressions in those.
libgl can be dlopened, glibc can't be. That is the problem. If libgl has some incompatibility, I can try to work around that. If glibc has some incompatibility, my executable won't even launch.
I agree with this article completely.
3 replies →
> then complain entirely about glibc
That’s because when you’re trying to ship a product glibc is one of the primary sources of pain.
If you have 3 hours or less of development experience on linux.