Comment by WD-42
4 hours ago
Did you hire a Linux release engineer? Or was the situation the typical team of devs maining macOS that have never heard the term “Wayland” before plus That One Guy who switched to Ubuntu last year and advocated for it?
There are companies that do this right. But it often requires a hire. Too many companies think they can just yolo it because Linux isn’t a serious OS or whatever and then are surprised when it doesn’t work out well.
> Did you hire a Linux release engineer
That's often a great idea!
But a full time hire? The GP's post implies that wouldn't make business sense for them, as even half a day occasionally on it is too much...
>> So your engineers spend a half day installing that in a VM and debugging it, but the problem is in upstream somewhere. The number of tickets with Linux issues keeps growing and each one is taking more time to debug, all for a number of customers that is so small you can’t justify doing it.
Of course an experienced Linux release engineer can do it faster and more reliably. That's probably the cheaper option. But the business still has to decide their Linux customer or user base is large enough, or strategically worth supporting, to justify the cost however they do it.
For many businesses even fractional Linux support is not justifiable for the small number of Linux users and support requests they're unable to handle. Though I can't imagine that being the case for Anthropic!
(Hint: This is one of the things I consult on, if anyone is looking to pay for quality Linux release engineering and platform testing. I have hundreds of historical and current Linux VMs, multiple architectures old and new (esp. x86, ARM and RISC-V), some of them embedded, fairly deep knowledge of how the kernel and libraries work together, and test harnesses. Also I test some compiled applications for portability across other OSes and architectures, including Windows, MS-DOS, MacOS, BSDs, SunOS, HP-UX, etc. going all the way back to the early Unix lineage.)
Even for those who do this right, some things change under your feet because OSS maintainers of kernel feature A want to stop supporting V1 of A when V2 has been out for a decade. But the features missing in V2 are supposed to be provided by userspace B - and they are yet to tackle the functionality altogether. So now your app will just have to regress in features. It is very easy to ship OSS code as a maintainer of a project, it is very difficult to keep up with Linux as a developer unless you stick to libc. There is no one source of truth with regards to how things should work, there is no one roadmap, and maintainers care a lot more about complexity than maintaining feature parity of backwards compatibility. I do not blame them, but then it is difficult to target linux. Much easier to support a platform with guarantees and a shared vision. Saying this as someone who has only used Linux at home for 20 years.
Thanks to the Linux kernel's extremely high backward compatibility, and virtually all the libraries being open source, you can ship old or frozen versions of libraries with your application if you have to. You can defensively set shipped binaries as fallbacks in the event the application is running on a newer system that dropped critical functionality, while using the distro version if that's more up to date and still has the functionality. You can do the same for auxiliary programs your application uses.
I agree that sticking to libc is most reliable, if you can. But the experience is poor if you do that for desktop applications.
There's no singular source of truth, but there's a de facto frontier of only a few mainstream distros, as well as upstream heads for your dependencies.
It's extra work, but there are systematic workarounds to the feature drift over time and the tendancy of some open source projects to aggressively deprecate older functionality and older system compatilbilty.
You can, to an extent, automate testing on newer versions of distros to be alerted when something no longer works, and often you can do this before the official distro release date.
Unfortunately even libc is not reliable. Unless it's a static build, Glibc is often broken (with symbol version errors) when trying to run a binary compiled on one distro on another distro, or an older version of the same distro. Static binaries have other problems, though work very well if the application is self contained and isn't a GUI.
One thing that I find works very compatibly, though, is OpenGL / Vulkan binary-compatibility across distros and versions. There was a lot of work done on making libGL something you can link to or dynamically load reliably and take it from there. The OpenGL extension spaghetti is an interesting problem from then on, but that's more to do with the individual user's GPU and GPU drivers, independent of the Linux distro or even which OS it's running on.
You cannot ship features that depend on cgroups v1. You may not ship features that depend on netlink attributes that exist on some distros, not others.
> You can defensively set shipped binaries as fallbacks in the event the application is running on a newer system that dropped critical functionality
Not if they're GPL licensed you can't. And that's a headache most commercial people do not want at all when trying to write software that's often for a marginal part of their audience anyway.
2 replies →