Comment by mananaysiempre
3 years ago
Surprisingly, that seems correct—a Flatpak bundle includes a glibc; though that only leaves me with more questions:
- On one hand, only one version of ld.so can exist in a single address space (duh). Glibc requires carnal knowledge of ld.so, thus only one version of glibc can exist in a single address space. In a Flatpak you have (?) to assume the system glibc is incompatible with the bundled one either way, thus you can’t assume you can load host libraries.
- On the other hand, a number of system services on linux-gnu depend on loading host libraries. Even if we ignore NSS (or exile it into a separate server process as it should have been in the first place), that leaves accelerated graphics: whether you use Wayland or X, ultimately an accelerated graphics driver amounts to a change in libGL and libEGL / libGLX (directly or through some sort of dispatch mechanism). These libraries require carnal knowledge of the kernel-space driver, thus emphatically cannot be bundled; but the previous point means that you can’t load them from the host system either.
- Modern toolkits basically live on accelerated graphics. Flatpak was created to distribute graphical applications built on modern toolkits.
- ... Wait, what?
There is no 'system' glibc. Linux doesn't care. The Linux kernel loads up the ELF interpreter specified in the ELF file based on the existing file namespace. If that ELF interpreter is the system one, then linux will likely remap it from existing page cache. If it's something else, linux will load it and then it will parse the remaining ELF sections. Linux kernel is incredibly stable ABI-wise. You can have any number of dynamic linkers happily co-existing on the machine. With Linux-based operating systems like NixOS, this is a normal day-to-day thing. The kernel doesn't care.
> These libraries require carnal knowledge of the kernel-space driver, thus emphatically cannot be bundled; but the previous point means that you can’t load them from the system either.
No they don't. The Linux kernel ABI doesn't really ever break. Any open-source driver shouldn't require any knowledge of internals from user-space. User-space may use an older version of the API, but it will still work.
> whether you use Wayland or X, ultimately an accelerated graphics driver amounts to a change in libGL and libEGL / libGLX (directly or through some sort of dispatch mechanism)
OpenGL is even more straightforwards because it is typically consumed as a dynamically loaded API, thus as long as the symbols match, it's fairly straightforwards to replace the system libGL.
I know, I both run NixOS and have made syscalls from assembly :) Sorry, slipped a bit in my phrasing. In the argument above, instead of “the system glibc” read “the glibc targeted by the compiler used for the libGL that corresponds to the graphics driver loaded into the running kernel”. (Unironically, the whole point of the list above was to avoid this sort of monster, but it seems I haven’t managed it.)
> No they don't. The Linux kernel ABI doesn't really ever break. Any open-source driver shouldn't require any knowledge of internals from user-space.
[laughs in Nvidia]
NVIDIA is not an open-source driver [1], and if you look in your dmesg logs, your kernel will complain about how it's tainted. That doesn't change the truth value about what I said about 'open-source' drivers.
[1] I think this may have changed very very recently.
This is all correct and I'd also add that ld.so doesn't need to have any special knowledge of glibc (or the kernel) in the first place. From the POV of ld.so, glibc is just another regular ELF shared object that uses the same features as everything else. There's nothing hard-coded in ld.so that loads libc.so.6 differently from anything else. And the only thing ld.so needs to know about the kernel is how to make a handful of system calls to open files and mmap things, and those system calls that have existed in Linux/Unix for eternity.
Needs to have? In an ideal world, probably not. Has and uses? Definitely. For one thing, they need to agree about userland ABI particulars like the arrangement of thread-local storage and so on, which have not stayed still since the System V days; but most importantly, as a practical matter, ld.so lives in the same source tree as glibc, exports unversioned symbols marked GLIBC_PRIVATE to it[1], and the contract between the two has always been considered private and unstable.
[1] https://sourceware.org/git/?p=glibc.git;a=blob;f=elf/Version...
> On one hand, only one version of ld.so can exist in a single address space (duh). Glibc requires carnal knowledge of ld.so, thus only one version of glibc can exist in a single address space.
Yes
> In a Flatpak you have (?) to assume the system glibc is incompatible with the bundled one either way, thus you can’t assume you can load host libraries.
Not exactly. You must assume that the host glibc is incompatible with the bundled one, that's right.
But that does not mean you cannot load host libraries. You can load them (provided you got them somehow inside the container namespace, including their dependencies) using the linker inside the container.
> hether you use Wayland or X, ultimately an accelerated graphics driver amounts to a change in libGL and libEGL / libGLX (directly or through some sort of dispatch mechanism).
In Wayland, your app tells the server to render a bitmap. How you got that bitmap rendered is up to you.
The optimization is that you send dma-buf handle instead of a bitmap. This is a kernel construct, not userspace driver one. This allows also cross-API app/compositor (i.e. Vulkan compositor and OpenGL app, or vice-versa). This also means you can use different version of the userspace driver with compositor than inside the container and they share the kernel driver.
> These libraries require carnal knowledge of the kernel-space driver, thus emphatically cannot be bundled; but the previous point means that you can’t load them from the host system either.
Yes and no; Intel and AMD user space drivers have to work with variety of kernel versions, so they cannot be too tight. Nvidia driver has tightly coupled user space and kernel space, but with the recent open-sourcing the kernel part, this will also change.
> but the previous point means that you can’t load them from the host system either.
You actually can -- bind mount that single binary into the container. You will use binary from the host, but load it using ld.so from inside container.
>> In a Flatpak you have (?) to assume the system glibc is incompatible with the bundled one either way, thus you can’t assume you can load host libraries.
> Not exactly. You must assume that the host glibc is incompatible with the bundled one, that's right.
> But that does not mean you cannot load host libraries. You can load them (provided you got them somehow inside the container namespace, including their dependencies) using the linker inside the container.
I meant that the glibcs are potentially ABI-incompatible both ways, not just that they’ll fight if you try to load both of them at once. Specifically, if the bundled (thus loaded) glibc is old, 2.U, and you try to load a host library wants a new frobnicate@GLIBC_2_V, V > U, you lose, right? I just don’t see any way around it.
>> These libraries require carnal knowledge of the kernel-space driver, thus emphatically cannot be bundled; but the previous point means that you can’t load them from the host system either.
> Yes and no; Intel and AMD user space drivers have to work with variety of kernel versions, so they cannot be too tight. Nvidia driver has tightly coupled user space and kernel space, but with the recent open-sourcing the kernel part, this will also change.
My impression of out-of-tree accelerated is mainly from fighting fglrx for the Radeon 9600 circa 2008, so extremely out of date. Intel is in-tree, so I’m willing to believe it has some degree of ABI stability, at least if an i915 blog post[1] is to be believed. Apparently AMD is also in-tree these days. Nvidia is binary-only, so the smart thing for them would probably be to build against an ancient Glibc so that it runs on everything.
But suppose the year is 2025, and a shiny new GPU architecture has come out, so groundbreaking no driver today can even lay down command buffers for it. The vendor is kind enough to provide an open-source driver that gets into every distro, and the userspace portion compiled against a distro-current Glibc ends up referencing an AVX-512 memcpy@GLIBC_3000 (or something).
I load a flatpak using Gtk3 GtkGLArea from 2015.
What happens?
[1] https://blog.ffwll.ch/2013/11/botching-up-ioctls.html
> I meant that the glibcs are potentially ABI-incompatible both ways, not just that they’ll fight if you try to load both of them at once. Specifically, if the bundled (thus loaded) glibc is old, 2.U, and you try to load a host library wants a new frobnicate@GLIBC_2_V, V > U, you lose, right? I just don’t see any way around it.
Yup. So the answer is to minimize the amount of loaded host libraries, ideally 0. If that cannot be done, the builder of that host library will have to make sure it is backward compatible.
> But suppose the year is 2025, and a shiny new GPU architecture has come out, so groundbreaking no driver today can even lay down command buffers for it. The vendor is kind enough to provide an open-source driver that gets into every distro, and the userspace portion compiled against a distro-current Glibc ends up referencing an AVX-512 memcpy@GLIBC_3000 (or something).
> I load a flatpak using Gtk3 GtkGLArea from 2015.
Ideally, that driver would be build as an extension of the runtime your flatpak uses. I.e. everything based on org.freedesktop.Platform (or derivatives like org.gnome.Platform and org.kde.Platform) has extensions maintained with appropriate Mesa and Nvidia user space drivers.
So if new open source driver, if it is not part of Mesa, or you are not using above mentioned runtimes, would need to be build against the 2015 runtime. The nice thing is, that the platforms have corresponding sdks, so it is not a problem targeting specific/old version.
Does graphics on Linux work by loading the driver into your process? I assumed it works via writing a protocol to shared memory in case of Wayland, or over a socket (or some byzantine shared memory stuff that is only defined in the Xorg source) in case of X11.
From my experience, if you have the kernel headers and have all the required options compiled into your kernel, then you can go really far back and build a modern glibc and Gtk+ Stack, and use a modern application on an old system. If you do some tricks with Rpath, everything is self-contained. I think it should work the other way around, with old apps on a new kernel + display server, as well.
So there are two parts to this: the app producing the image in the application window and then the windowing system combining multiple windows together to form the final image you see on screen.
The former gets done in process (using e.g. GL/vulkan) and then that final image gets passed onto the windowing system which is a separate process and could run outside the container.
As an aside, with accelerated graphics you mostly pass a file descriptor to the GPU memory containing the image, rather than mucking around with traditional shared memory.
Does graphics on Linux work by loading the driver into your process?
Yes, it's called Direct Rendering (DRI) and it allows apps to drive GPUs with as little overhead as possible. The output of the GPU goes into the shared memory so that the compositor can see it.