Being an upstream maintainer is incredibly under-appreciated. It’s an unfathomably hard, and somewhat thankless, job (at least if you do it well). A friend of mine was in a cab with Ted Ts’o at a conference and he was reviewing patches on his phone to keep up with the workload (or maybe he was bored who knows).
Despite incredible effort from maintainers, getting necessary changes into Linux can take forever. In the subsystem I depend on (and occasionally contribute to directly) it’s kinda assumed it will take at least a year (probably two) for any substantial project to get merged. This continuously disappoints PMs and Leadership. A lot of people, understandably, chafe against this lack of agility.
OTOH, I’ve been on the other side of kernel bugs. Most recently, a memory arithmetic bug was causing corruption, and took my team at least an engineer year to track down. This makes me quite sympathetic to maintainers demands for quality.
I’ve also been on the other side of the calibration discussions where Open Source work goes under appreciated. The irony never stops (“They won’t merge our patches!” “Are you having your engineers review theirs?”). That and the raw pipeline issues for maintainers (it takes a lot of experience to be a maintainer, which implies spending a lot of a bright engineer’s time on reviewing and contributing upstream to things unrelated to immediate priorities).
a bug taking a year to track down is a negative indicator of the quality of project maintenance, not the person who contributed the bug, whether it's due the code itself or the tooling and testing environments available to verify such important issues.
This isn't wrong per se, but rather, it lacks concrete recommendations for what should be done differently.
I would love to see Linux thoroughly and meaningfully tested. For some parts it's just... hard. (If anyone wants to get their start writing kernel code, have a crack at writing some self-tests for a component that looks complicated. The relevant maintainer will probably be excited to see literally anyone writing tests.)
For this particular bug, the cheapest spot to catch the issue would have been code review. In a normal code base, the next cheapest would have been unit testing, though, in this situation, that may not have caught it given that the underlying bug required someone to break the contract of a function (one part of Linux broke the contract of another. Why did it not BUG_ON for that...).
Eliminating the class of issue required fairly invasive forms of introspection on VMs running a custom module. Sure, we did that... eventually.
Finding it originally required stumbling on a distro of Linux that accidentally manifested the corruption visibly (about once per 50ish 30 minute integration test runs, which is pretty frequently in the scheme of corruption bugs).
The factors at play are all obvious. The old-school guys want to keep things old-school. The new-school guys want to make things better in a new way. Has there been any new rewrite without a BFDL who himself is of the new school? The vim/neovim schism happened and perhaps that's how it ends. I personally like Neovim and I'm glad to be in backers.md and it's a tremendously larger amount of work than to have just changed vim. But c'est la vie.
Egcs vs. gcc was a big deal back in the day and in the end we ended up with gcc by the egcs guys and that was it. When you win, everyone forgets the 'drama' existed. When you lose, everyone remembers you as just the drama guy. RMS had the drama label for decades. Things are not even different. They're the same again. It's like when you'd buy those Chinese NES dupes and they'd have 999 levels of Mario but half the levels would be the same but with different colour bricks. Isomorphic to original but distinct. That's this story.
> didn't the good egcs stuff get merged after all?
EGCS took over as mainline, and what good stuff there was in the old mainline got merged into it. But it took sustaining the fork for years to make that happen.
> The new-school guys want to make things better in a new way.
The new guys definitely want to make things different, but it seems there is a lot of debate over whether it will actually be better. Really, they should just write their own their kernel. If rust is really that much better, they'll will.
There are already kernels written in Rust. Telling them to go write new kernels in Rust is like telling people working on a new audio workstation that they should write a package manager instead. A new kernel that they cannot practically use does not suit their needs. The point is to use Rust where it can suit their needs.
This seems quite ironic to say when the whole drama started with Christoph not even looking at the patches long enough to see what directory they were in before rejecting them.
What happened here? Years ago Linus was talking about how he thought positively about Rust in the Kernel in the future if the kinks could be worked out. Now a group of people have built out a set of drivers which are working great, well tested and integrated, and one maintainer has decided they just don't want to merge it so the whole project is indefinitely stalled.
I'd be pretty upset if I was working on Asahi since the Linux project has basically bait and switched them after an enormous amount of work has been invested.
> The old-school guys want to keep things old-school.
You are missing the whole point here. The kernel is a survival epic amongst millions of other failed projects. You don't get to tell the old captain and its lieutenants how to nail the planks and helm the ship when you just went pass the Titanic and Britannic wrecks because metal is so cool.
They're "old-school" because they have to be. Engineers will excrete their pet project and then leave and now they will have to support it. They are mean because that's the only "power" they have, as is explained in the post.
> This is not the world of the systems hacker. When you debug a
distributed system or an OS kernel, you do it Texas-style. You
gather some mean, stoic people, people who have seen things
die, and you get some primitive tools, like a compass and a
rucksack and a stick that’s pointed on one end, and you walk
into the wilderness and you look for trouble, possibly while
using chewing tobacco. As a systems hacker, you must be pre-
pared to do savage things, unspeakable things, to kill runaway
threads with your bare hands, to write directly to network
ports using telnet and an old copy of an RFC that you found in
the Vatican. When you debug systems code, there are no high-
level debates about font choices and the best kind of turquoise,
because this is the Old Testament, an angry and monochro-
matic world, and it doesn’t matter whether your Arial is Bold
or Condensed when people are covered in boils and pestilence
and Egyptian pharaoh oppression. HCI people discover bugs
by receiving a concerned email from their therapist. Systems
people discover bugs by waking up and discovering that their
first-born children are missing and “ETIMEDOUT ” has been
written in blood on the wall.
What is despair? I have known it—hear my song. Despair is
when you’re debugging a kernel driver and you look at a mem-
ory dump and you see that a pointer has a value of 7. THERE IS
NO HARDWARE ARCHITECTURE THAT IS ALIGNED ON
7. Furthermore, 7 IS TOO SMALL AND ONLY EVIL CODE
WOULD TRY TO ACCESS SMALL NUMBER MEMORY.
Misaligned, small-number memory accesses have stolen
decades from my life. The only things worse than misaligned,
small-number memory accesses are accesses with aligned buf-
fer pointers, but impossibly large buffer lengths. Nothing ruins
a Friday at 5 P.M. faster than taking one last pass through the
log file and discovering a word-aligned buffer address, but a
buffer length of NUMBER OF ELECTRONS IN THE UNI-
VERSE.
A fundamental problem here not yet discussed directly here is how few maintainers there really are for a software project of this magnitude and importance.
Further, the fact that so many of those maintainers are purely on volunteer time.
Now it is certainly somewhat the fault of the maintainers themselves for turning off thousand if not tens of thousands of eager, well-intentioned wannabe contributors over the decades, if not through their attitudes and lack of interpersonal skills, then through impenetrable build systems and hostility towards ergonomic changes.
But forget the eager amateurs - it is unconscionable that major technology companies & cloud providers don't each have damn near an army helping out with Linux and similar technologies - even the parts that do not directly benefit them! - instead of just shoving it into servers so they can target ads for cheap plastic crap 0.000000001% better than they did last week.
Is that really the right statistic? Seems like the relevant one would be the number of maintainers whose maintenance work is company-funded. (Ex, I'd imagine it would be quite bad if most contributions were from company-funded developers but had to be upstreamed by non-company-funded volunteers.)
> But what isn't appreciated, is that it is precisely because people who are long-term members of the community are trusted to stick around and will support code that the have sponsored.
Ok. But if you make it too difficult for new developers to contribute, and the experience too stressful, then no one new will stick around to be long time maintainers, and when the old ones retire or die, there won't be anyone to replace them.
> One of the things which gets very frustrating from the maintainer's
perspective is development teams that are only interested in their pet
feature, and we know, through very bitter experience, that 95+% of
the time, once the code is accepted, the engineers which contribute
the code will disappear, never to be seen again.
This was painful for me to read as someone who has seen how corporations think about “Open Source” - he isn’t wrong at all
He's not wrong, but he hasn't addressed the problem:
Some maintainers are rejecting changes as a way to block a project they disagree with - ie. there is no path forward for the contributor. I wouldn't assume this normally, but they're not hiding this fact, it's self proclaimed.
On the other hand, Linus has been largely in favour of the R4L project, and gave the green light for it to go ahead.
The Linux kernel maintainers need to figure out among themselves whether they want to allow Rust to be introduced or not, and if so under what constraints. If they can't come to an agreement, they're wasting everyone's time.
Once that happens, either the project is canned, or people can stop arguing over if these changes should be upstreamed, and start arguing over how instead, and that's a lot more productive.
It's not the maintainers job to make your project happen.
It's yours.
If you need someone to do free work but you can't convince them then you either get their boss to tell them to do it, you take over their job, or - the one thing that no one under 30 ever seems to do - fork and do the work yourself without anyone stopping you.
IIRC Linus has been in favor conditioned on the agreement of the respective subsystem maintainers. Meaning, he’s not deciding over their heads. And introduction of Rust can be handled differently from subsystem to subsystem.
That doesn't exactly strike me as a bad thing? Isn't that sort of the point of open source? Lots of people working on the things that interest them and through a diversity of interests arriving at a useful project?
> the engineers which contribute the code will disappear
From the other side, as a very occasional contributor, I'd actually want to deal with fixes or reviews around the code I contribute.
But it's usually edge cases on otherwise stable and mature libraries, so hopefully it probably won't happen more than say once in a decade. If I got a mention on a PR I'd reappear, but that doesn't sound like the standard way, I never got involved again on anything submitted.
I feel like either the maintainers are doing an incredibly good job at vetting the PRs, or got used to deal with the aftermath another way and just don't need the original code submitter to reappear most of the time ?
I empathize but we need to have people take over these initiatives and refactor them into something easier the maintain. Not saying introduce a new language or anything but change how we fundamentally look at “The Kernel”. I think reducing scope and making it so hardware providers must maintain their drivers is a good start. If your toolchain doesn’t suffice, create a new tool.
If rust is so much better, create a new kernel in rust and force a paradigm shift. I think we can do better than bicker and fight over it and post email chains about it. Bring solutions. Tech debt is just OpEx to everyone else.
> create a new kernel in rust and force a paradigm shift.
And then what?
A kernel by itself isn't very useful. Even if your kernel is somehow superior in every way, you need software to target it, so to be a successful replacement for linux you basically have to be completely compatible with any software that currently runs on linux, which not only means a massive amount of work just to keep up with changes in linux, but constraints on your own design.
And then there is hardware support. If you are a fledgling project, how do you get hardware vendors to write drivers for you?
>Bring solutions. Tech debt is just OpEx to everyone else.
What if there was a rust compiler to C, that produced like readable, compliant C code for the kernel. And then developers that want to work in rust can publish their original rust code to some third party location, so the reviewers have no idea if the C code they receive it was originally written in rust or not.
And what happens if somebody starts modifying that C code? Even a one way translator like you describe could be a challenge, making it two way seems close to impossible.
It's a cool idea but it's licensed MIT instead of GPL or something copyleft so I won't contribute to it.
Rust-only ecosystem would be pretty cool though. It may be worth just forking linux for the sake of compatibility, and keeping the license going. I don't see a future otherwise.
I'd also like to see better compiler diversity. Maybe once gccrs rolls around we will see different attitudes around rust emerge, compared to C/C++ which have more distributed development.
If comments as benign as "thin blue line" causes fragile entryist/activists to flee, I say Ted and the kernel team are doing the right thing. Projects as critical as the Linux kernel shouldn't be battlegrounds for the grievance of the week, nor should they be platforms for proselytizing. Others like him leave long paths of destruction in their wake. Lots of projects have been turned upsidedown by the drama they seem to bring with them everywhere. The salient point is contributors need to be more than "drive by" submitters for their pet projects. This isn't specific to Rust in the kernel, look at how much of an uphill battle bcachefs was/is.
gcc vs. egcs, emacs vs xemacs, bsd vs bsd, bsd vs att, etc, etc, etc.
FOSS evolves past its chokepoints by forking.
So, a credible group of Rustaceans and their backers need to come up with a plan to do it.
It doesn't have to be antagonistic -- it's exploratory. If it works out well, it's a lot easier to adopt once the imagined issues are resolved or evaporated.
Subsystem by subsystem would be my suggestion. And just do it. And sooner or later, it will be (a) good enough that it's pulled into -next, or (b) they'll give up because it's not worth the effort.
I'd be pretty surprised if, for instance, one of Google/Amazon/Meta/Microsoft/Cloudflare/Netflix/etc wasn't interested in an ABI-compatible kernel written in Rust. Get a few biggish backers, and LF would possibly even adopt it.
There was a reasonable discussion about Rust, and then T'so drops the bombshell:
"I suspect part of the problem here is you're trying to convince everyone to switch over to the religion as promulgated by Rust and the reality is that ain't going to happen because we have 50 plus file systems in Linux they will not all be instantaneously converted over to rust" [26:05]
He pretty much shouts at the Rust developers, denigrates their efforts and when they explain what they are trying to do which is to understand the semantic changes.
And then T'so then starts yelling at him even more. He wonders why people are not happy with his attitude - then perhaps he should consider how he communicates. The Rust guy is trying to be constructive and T'so was not.
yeah he managed to try to sound reasonable and measured for most of that, then:
> including an upstream language community which refuses to make any kind of backwards compatibility guarantees, and which is actively hostile to a second Rust compiler implementation (I suspect because it might limit their ability to make arbitrary backwards-incompatble language changes).
with the full context of all of this it is quite obvious this is just a classic case of a software engineer being an antisocial jerk with strange vendettas in their head and who thinks the strange personal opinions they came up with in their basement are unassailable facts
>> including an upstream language community which refuses to make any kind of backwards compatibility guarantees, and which is actively hostile to a second Rust compiler implementation (I suspect because it might limit their ability to make arbitrary backwards-incompatble language changes).
The guy is a poster who just can't help himself! Both points are demonstrably false.
Huh, then don't listen. So far no clear answer why Rust people are suffering daily humiliation at the hand of C coders instead of just writing their own kernel.
Calling yourself the “thin blue line” like a reference to police really doesn’t seem like a way to have an enduring community. Police don’t serve the community; I guess they think in the same way then.
>The thin blue line U.S. flag has been banned by some police departments in the United States for its associations with ideologies described as "undemocratic, racist, and bigoted."
>According to a 2018 law review article, "thin blue line" also refers to an unwritten code of silence used to cover up police misconduct, also known as the blue wall of silence, a term dating back to 1978
Seems like ignorance of the implications, not any desire to a tie themselves to a political stance. I think the “assume positive intent” principle needs to apply here.
Law review articles are largely written by students. They are not authoritative.
Do you think that Rowan Atkinson was dogwhistling about undemocratic, racist, bigoted police violence when he named his police sitcom "The Thin Blue Line" or do you think perhaps your dislike of the police has led to you associating negative attributes with anyone that happens to use any phrase related to the police?
Where I am from, the police is a respected, largely unarmed institution. Imperfect, as all institutions are. American left-political dislike of law enforcement isn't universal and most people don't have instant negative mental associations with everything police-related...
Within capitalist societies police protect the interests of the state, which itself exists to serve and benefit the capitalist class. Police "maintain order" to ensure the proletariat continues to submit to the state and participate in the capitalist machine, and they commit violence against anyone who does otherwise.
Look into the history and meaning of “thin blue line.” It’s an endorsement of the us vs. them mindset prevalent in American policing that is the source of many (most? all?) police scandals.
They are largely there to enforce the law and investigate transgressions of it. If you've ever reported a theft lately you'd know that they are not resourced - at least around here - to put any priority on protecting anyone's property. They have their hands full dealing with domestics and drug and alcohol related offending.
why cannot more people learn about git and branching
you want your merges in? patch it in your own branch
the real crux of the issue is the quality of "the offical version", the real issue, then, is about official branding.
I remember back in the day there were multiple kernel versions by different mantainers.... then again I'm probably missing the forest for the trees or something
Because forked kernels is how we ended up with all those SBCs which could only run some ancient version of linux. If you can't mainline it, you are committing to maintaining a fork forever, and the more changes you require, the more maintenance that fork will be.
The Linux kernel quite intentionally makes maintaining a fork a pain in the ass to pressure vendors to mainline their drives. But now mainline maintainers are refusing to accept those drivers. So it's unlikely we will ever see Macbooks properly supported on Linux.
But the Rust advocates are already telling everyone that they will handle all maintenance of rust code in the kernel and existing maintainers won't have to do a thing. If that was true it wouldn't matter if the rust code is in a separate tree.
I feel like this problem should have a technical solution.
maybe a VCS good enough that this isn't as big an issue? (along the lines of pijul?) or
maybe something from formal verification methods to better enforce or more clearly explain why some drivers cannot be accepted. then again, I don't think the computer science is quite there yet
This phrase has been largely co-opted by US white nationalist/fascist groups (similar to how Pepe the Frog was) but that is a relatively recent development; the phrase has been in use for a long time before that.
I think that there's potentially a couple of things at play here:
Taken literally, the idea of a "thin blue line" protecting society from chaos has some authoritarian overtones.
It's a slogan that tends to be used to simply dismiss problems like police brutality. That is to say, it comes across less like a difference in policy position and more like indifference to the suffering of others.
The political faction that it's associated with, i.e. MAGA, is currently in the process of attempting to purge anyone with a worldview that they dislike from public society.
All of those factors creates a situation where it is difficult for those who use signifiers like "thin blue line" to practically operate simply as "people who happen to be conservative" as opposed to "people who represent an irreconcilable threat to people they disagree with".
The burden of proof isn't on maintainers. Recognizing new, untried code invariably means bugs and instability to work through, often for questionable gains, is a mark of an experienced developer. Work your way through enough industry hype cycles and you'll see, new shit just causes new and usually weirder problems than whatever it was intended to replace.
Related: https://news.ycombinator.com/item?id=43036904
Thanks! Macroexpanded:
Resigning as Asahi Linux project lead - https://news.ycombinator.com/item?id=42972062 - Feb 2025 (1015 comments)
Next resignation:
Me stepping down as a nouveau kernel maintainer – https://lists.freedesktop.org/archives/nouveau/2025-February...
Being an upstream maintainer is incredibly under-appreciated. It’s an unfathomably hard, and somewhat thankless, job (at least if you do it well). A friend of mine was in a cab with Ted Ts’o at a conference and he was reviewing patches on his phone to keep up with the workload (or maybe he was bored who knows).
Despite incredible effort from maintainers, getting necessary changes into Linux can take forever. In the subsystem I depend on (and occasionally contribute to directly) it’s kinda assumed it will take at least a year (probably two) for any substantial project to get merged. This continuously disappoints PMs and Leadership. A lot of people, understandably, chafe against this lack of agility.
OTOH, I’ve been on the other side of kernel bugs. Most recently, a memory arithmetic bug was causing corruption, and took my team at least an engineer year to track down. This makes me quite sympathetic to maintainers demands for quality.
I’ve also been on the other side of the calibration discussions where Open Source work goes under appreciated. The irony never stops (“They won’t merge our patches!” “Are you having your engineers review theirs?”). That and the raw pipeline issues for maintainers (it takes a lot of experience to be a maintainer, which implies spending a lot of a bright engineer’s time on reviewing and contributing upstream to things unrelated to immediate priorities).
a bug taking a year to track down is a negative indicator of the quality of project maintenance, not the person who contributed the bug, whether it's due the code itself or the tooling and testing environments available to verify such important issues.
This isn't wrong per se, but rather, it lacks concrete recommendations for what should be done differently.
I would love to see Linux thoroughly and meaningfully tested. For some parts it's just... hard. (If anyone wants to get their start writing kernel code, have a crack at writing some self-tests for a component that looks complicated. The relevant maintainer will probably be excited to see literally anyone writing tests.)
For this particular bug, the cheapest spot to catch the issue would have been code review. In a normal code base, the next cheapest would have been unit testing, though, in this situation, that may not have caught it given that the underlying bug required someone to break the contract of a function (one part of Linux broke the contract of another. Why did it not BUG_ON for that...).
Eliminating the class of issue required fairly invasive forms of introspection on VMs running a custom module. Sure, we did that... eventually.
Finding it originally required stumbling on a distro of Linux that accidentally manifested the corruption visibly (about once per 50ish 30 minute integration test runs, which is pretty frequently in the scheme of corruption bugs).
2 replies →
The factors at play are all obvious. The old-school guys want to keep things old-school. The new-school guys want to make things better in a new way. Has there been any new rewrite without a BFDL who himself is of the new school? The vim/neovim schism happened and perhaps that's how it ends. I personally like Neovim and I'm glad to be in backers.md and it's a tremendously larger amount of work than to have just changed vim. But c'est la vie.
Egcs vs. gcc was a big deal back in the day and in the end we ended up with gcc by the egcs guys and that was it. When you win, everyone forgets the 'drama' existed. When you lose, everyone remembers you as just the drama guy. RMS had the drama label for decades. Things are not even different. They're the same again. It's like when you'd buy those Chinese NES dupes and they'd have 999 levels of Mario but half the levels would be the same but with different colour bricks. Isomorphic to original but distinct. That's this story.
if your lens is old-bad vs new-good, you are blind to merits.
didn't the good egcs stuff get merged after all?
> didn't the good egcs stuff get merged after all?
EGCS took over as mainline, and what good stuff there was in the old mainline got merged into it. But it took sustaining the fork for years to make that happen.
> The new-school guys want to make things better in a new way.
The new guys definitely want to make things different, but it seems there is a lot of debate over whether it will actually be better. Really, they should just write their own their kernel. If rust is really that much better, they'll will.
Well I certainly don't use linux because it's written in C. Who would?
Edit: This seems like more of an indication of a culture that lacks effective conflict resolution than any kind of technical question.
There are already kernels written in Rust. Telling them to go write new kernels in Rust is like telling people working on a new audio workstation that they should write a package manager instead. A new kernel that they cannot practically use does not suit their needs. The point is to use Rust where it can suit their needs.
2 replies →
>The old-school guys want to keep things old-school. The new-school guys want to make things better in a new way
The new school guys greatly underappreciate the wisdom of why and instead try to change things without first understanding.
This seems quite ironic to say when the whole drama started with Christoph not even looking at the patches long enough to see what directory they were in before rejecting them.
4 replies →
What happened here? Years ago Linus was talking about how he thought positively about Rust in the Kernel in the future if the kinks could be worked out. Now a group of people have built out a set of drivers which are working great, well tested and integrated, and one maintainer has decided they just don't want to merge it so the whole project is indefinitely stalled.
I'd be pretty upset if I was working on Asahi since the Linux project has basically bait and switched them after an enormous amount of work has been invested.
1 reply →
> The old-school guys want to keep things old-school.
You are missing the whole point here. The kernel is a survival epic amongst millions of other failed projects. You don't get to tell the old captain and its lieutenants how to nail the planks and helm the ship when you just went pass the Titanic and Britannic wrecks because metal is so cool.
They're "old-school" because they have to be. Engineers will excrete their pet project and then leave and now they will have to support it. They are mean because that's the only "power" they have, as is explained in the post.
I'll leave you with this quote from "The Night Watch" by James Mickens (https://www.usenix.org/system/files/1311_05-08_mickens.pdf)
> This is not the world of the systems hacker. When you debug a distributed system or an OS kernel, you do it Texas-style. You gather some mean, stoic people, people who have seen things die, and you get some primitive tools, like a compass and a rucksack and a stick that’s pointed on one end, and you walk into the wilderness and you look for trouble, possibly while using chewing tobacco. As a systems hacker, you must be pre- pared to do savage things, unspeakable things, to kill runaway threads with your bare hands, to write directly to network ports using telnet and an old copy of an RFC that you found in the Vatican. When you debug systems code, there are no high- level debates about font choices and the best kind of turquoise, because this is the Old Testament, an angry and monochro- matic world, and it doesn’t matter whether your Arial is Bold or Condensed when people are covered in boils and pestilence and Egyptian pharaoh oppression. HCI people discover bugs by receiving a concerned email from their therapist. Systems people discover bugs by waking up and discovering that their first-born children are missing and “ETIMEDOUT ” has been written in blood on the wall. What is despair? I have known it—hear my song. Despair is when you’re debugging a kernel driver and you look at a mem- ory dump and you see that a pointer has a value of 7. THERE IS NO HARDWARE ARCHITECTURE THAT IS ALIGNED ON 7. Furthermore, 7 IS TOO SMALL AND ONLY EVIL CODE WOULD TRY TO ACCESS SMALL NUMBER MEMORY. Misaligned, small-number memory accesses have stolen decades from my life. The only things worse than misaligned, small-number memory accesses are accesses with aligned buf- fer pointers, but impossibly large buffer lengths. Nothing ruins a Friday at 5 P.M. faster than taking one last pass through the log file and discovering a word-aligned buffer address, but a buffer length of NUMBER OF ELECTRONS IN THE UNI- VERSE.
The reasoning Linus himself gives for greenlighting Rust is, among other things, to avoid stagnation. So OP's description seems more apt than yours.
https://www.youtube.com/watch?v=OvuEYtkOH88&t=367s
3 replies →
A fundamental problem here not yet discussed directly here is how few maintainers there really are for a software project of this magnitude and importance. Further, the fact that so many of those maintainers are purely on volunteer time.
Now it is certainly somewhat the fault of the maintainers themselves for turning off thousand if not tens of thousands of eager, well-intentioned wannabe contributors over the decades, if not through their attitudes and lack of interpersonal skills, then through impenetrable build systems and hostility towards ergonomic changes.
But forget the eager amateurs - it is unconscionable that major technology companies & cloud providers don't each have damn near an army helping out with Linux and similar technologies - even the parts that do not directly benefit them! - instead of just shoving it into servers so they can target ads for cheap plastic crap 0.000000001% better than they did last week.
> Further, the fact that so many of those maintainers are purely on volunteer time.
Greg pointed out in that email thread that:
> over 80% of the contributions come from company-funded developers. [1]
[1] https://lore.kernel.org/lkml/2025020738-observant-rocklike-7...
Is that really the right statistic? Seems like the relevant one would be the number of maintainers whose maintenance work is company-funded. (Ex, I'd imagine it would be quite bad if most contributions were from company-funded developers but had to be upstreamed by non-company-funded volunteers.)
1 reply →
> major technology companies & cloud providers don't each have ...
... they have, and they are selling that as premium. it's the classic "open core" model for the cloud era.
> But what isn't appreciated, is that it is precisely because people who are long-term members of the community are trusted to stick around and will support code that the have sponsored.
Ok. But if you make it too difficult for new developers to contribute, and the experience too stressful, then no one new will stick around to be long time maintainers, and when the old ones retire or die, there won't be anyone to replace them.
I tried to help with the Linux kernel once in 2001 or so. I decided my calm was worth more than dealing with abrasive kernel devs.
Same thing happening with ham radio.
The whole community shows symptoms of organizational sabotage.
https://files.catbox.moe/u6rold.png
Sabotage or just the inevitable ways that organizations tend to decay (not that they have to, but if they do it tends to look a certain way).
> One of the things which gets very frustrating from the maintainer's perspective is development teams that are only interested in their pet feature, and we know, through very bitter experience, that 95+% of the time, once the code is accepted, the engineers which contribute the code will disappear, never to be seen again.
This was painful for me to read as someone who has seen how corporations think about “Open Source” - he isn’t wrong at all
He's not wrong, but he hasn't addressed the problem:
Some maintainers are rejecting changes as a way to block a project they disagree with - ie. there is no path forward for the contributor. I wouldn't assume this normally, but they're not hiding this fact, it's self proclaimed.
On the other hand, Linus has been largely in favour of the R4L project, and gave the green light for it to go ahead.
The Linux kernel maintainers need to figure out among themselves whether they want to allow Rust to be introduced or not, and if so under what constraints. If they can't come to an agreement, they're wasting everyone's time.
Once that happens, either the project is canned, or people can stop arguing over if these changes should be upstreamed, and start arguing over how instead, and that's a lot more productive.
It's not the maintainers job to make your project happen.
It's yours.
If you need someone to do free work but you can't convince them then you either get their boss to tell them to do it, you take over their job, or - the one thing that no one under 30 ever seems to do - fork and do the work yourself without anyone stopping you.
26 replies →
IIRC Linus has been in favor conditioned on the agreement of the respective subsystem maintainers. Meaning, he’s not deciding over their heads. And introduction of Rust can be handled differently from subsystem to subsystem.
That doesn't exactly strike me as a bad thing? Isn't that sort of the point of open source? Lots of people working on the things that interest them and through a diversity of interests arriving at a useful project?
> the engineers which contribute the code will disappear
From the other side, as a very occasional contributor, I'd actually want to deal with fixes or reviews around the code I contribute.
But it's usually edge cases on otherwise stable and mature libraries, so hopefully it probably won't happen more than say once in a decade. If I got a mention on a PR I'd reappear, but that doesn't sound like the standard way, I never got involved again on anything submitted.
I feel like either the maintainers are doing an incredibly good job at vetting the PRs, or got used to deal with the aftermath another way and just don't need the original code submitter to reappear most of the time ?
Am I missing some bigger part of it ?
The problem is aging code sours like milk.
I empathize but we need to have people take over these initiatives and refactor them into something easier the maintain. Not saying introduce a new language or anything but change how we fundamentally look at “The Kernel”. I think reducing scope and making it so hardware providers must maintain their drivers is a good start. If your toolchain doesn’t suffice, create a new tool.
If rust is so much better, create a new kernel in rust and force a paradigm shift. I think we can do better than bicker and fight over it and post email chains about it. Bring solutions. Tech debt is just OpEx to everyone else.
> create a new kernel in rust and force a paradigm shift.
And then what?
A kernel by itself isn't very useful. Even if your kernel is somehow superior in every way, you need software to target it, so to be a successful replacement for linux you basically have to be completely compatible with any software that currently runs on linux, which not only means a massive amount of work just to keep up with changes in linux, but constraints on your own design.
And then there is hardware support. If you are a fledgling project, how do you get hardware vendors to write drivers for you?
No part of this explains why Linux should be changed. It just means the new project has a lot of work to do.
>Bring solutions. Tech debt is just OpEx to everyone else.
What if there was a rust compiler to C, that produced like readable, compliant C code for the kernel. And then developers that want to work in rust can publish their original rust code to some third party location, so the reviewers have no idea if the C code they receive it was originally written in rust or not.
The output of a compiler produce unidiomatic code. And if the C gets edited you need to back port to Rust which is more work and may be impossible.
Try what you said with even well align languages like TS and JS and it would be hard to work with.
And what happens if somebody starts modifying that C code? Even a one way translator like you describe could be a challenge, making it two way seems close to impossible.
The problem is that the C guys don't want to define certain semantics. Currently "it works", but Rust wants invariants.
> making it so hardware providers must maintain their drivers is a good start
How do you propose that happens?
There is redox os
It's a cool idea but it's licensed MIT instead of GPL or something copyleft so I won't contribute to it.
Rust-only ecosystem would be pretty cool though. It may be worth just forking linux for the sake of compatibility, and keeping the license going. I don't see a future otherwise.
I'd also like to see better compiler diversity. Maybe once gccrs rolls around we will see different attitudes around rust emerge, compared to C/C++ which have more distributed development.
7 replies →
More shots fired at Linus (justifiably so) https://lore.kernel.org/lkml/6e3e26c7-002a-49c6-a3a8-1cc1b89...
"Shots fired" from a random anonymous person? Everyone can send their rant to LKML, that doesnt mean anything.
> I'm just a Linux user watching this shitshow unfold
Yet he can't even put up his real name. Pathetic
If comments as benign as "thin blue line" causes fragile entryist/activists to flee, I say Ted and the kernel team are doing the right thing. Projects as critical as the Linux kernel shouldn't be battlegrounds for the grievance of the week, nor should they be platforms for proselytizing. Others like him leave long paths of destruction in their wake. Lots of projects have been turned upsidedown by the drama they seem to bring with them everywhere. The salient point is contributors need to be more than "drive by" submitters for their pet projects. This isn't specific to Rust in the kernel, look at how much of an uphill battle bcachefs was/is.
gcc vs. egcs, emacs vs xemacs, bsd vs bsd, bsd vs att, etc, etc, etc.
FOSS evolves past its chokepoints by forking.
So, a credible group of Rustaceans and their backers need to come up with a plan to do it.
It doesn't have to be antagonistic -- it's exploratory. If it works out well, it's a lot easier to adopt once the imagined issues are resolved or evaporated.
Subsystem by subsystem would be my suggestion. And just do it. And sooner or later, it will be (a) good enough that it's pulled into -next, or (b) they'll give up because it's not worth the effort.
I'd be pretty surprised if, for instance, one of Google/Amazon/Meta/Microsoft/Cloudflare/Netflix/etc wasn't interested in an ABI-compatible kernel written in Rust. Get a few biggish backers, and LF would possibly even adopt it.
> FOSS evolves past its chokepoints by forking.
Unless you've been already invited into the project?
Unless I’m mistaking things, there’s still multiple roadblocks?
Just because some people have invited you in doesn’t mean everyone is welcoming.
1 reply →
I would be more likely to want to listen to T'so if he hadn't done the following:
https://www.youtube.com/watch?v=WiPp9YEBV0Q&t=1529s
There was a reasonable discussion about Rust, and then T'so drops the bombshell:
"I suspect part of the problem here is you're trying to convince everyone to switch over to the religion as promulgated by Rust and the reality is that ain't going to happen because we have 50 plus file systems in Linux they will not all be instantaneously converted over to rust" [26:05]
He pretty much shouts at the Rust developers, denigrates their efforts and when they explain what they are trying to do which is to understand the semantic changes.
And then T'so then starts yelling at him even more. He wonders why people are not happy with his attitude - then perhaps he should consider how he communicates. The Rust guy is trying to be constructive and T'so was not.
yeah he managed to try to sound reasonable and measured for most of that, then:
> including an upstream language community which refuses to make any kind of backwards compatibility guarantees, and which is actively hostile to a second Rust compiler implementation (I suspect because it might limit their ability to make arbitrary backwards-incompatble language changes).
with the full context of all of this it is quite obvious this is just a classic case of a software engineer being an antisocial jerk with strange vendettas in their head and who thinks the strange personal opinions they came up with in their basement are unassailable facts
>> including an upstream language community which refuses to make any kind of backwards compatibility guarantees, and which is actively hostile to a second Rust compiler implementation (I suspect because it might limit their ability to make arbitrary backwards-incompatble language changes).
The guy is a poster who just can't help himself! Both points are demonstrably false.
Maybe they're tired of the Rust Evangelism Strike Force.
Which means they can't explain interface semantics?
Huh, then don't listen. So far no clear answer why Rust people are suffering daily humiliation at the hand of C coders instead of just writing their own kernel.
Ted's a great guy. I haven't interacted with him for over 20 years, but IMHO, he's a rock star.
Calling yourself the “thin blue line” like a reference to police really doesn’t seem like a way to have an enduring community. Police don’t serve the community; I guess they think in the same way then.
I agree with you, and the reference _is_ politically loaded (https://en.wikipedia.org/wiki/Thin_blue_line):
>The thin blue line U.S. flag has been banned by some police departments in the United States for its associations with ideologies described as "undemocratic, racist, and bigoted."
>According to a 2018 law review article, "thin blue line" also refers to an unwritten code of silence used to cover up police misconduct, also known as the blue wall of silence, a term dating back to 1978
Seems like ignorance of the implications, not any desire to a tie themselves to a political stance. I think the “assume positive intent” principle needs to apply here.
9 replies →
Law review articles are largely written by students. They are not authoritative.
Do you think that Rowan Atkinson was dogwhistling about undemocratic, racist, bigoted police violence when he named his police sitcom "The Thin Blue Line" or do you think perhaps your dislike of the police has led to you associating negative attributes with anyone that happens to use any phrase related to the police?
Where I am from, the police is a respected, largely unarmed institution. Imperfect, as all institutions are. American left-political dislike of law enforcement isn't universal and most people don't have instant negative mental associations with everything police-related...
12 replies →
Police can serve the community - but people who use the reference "thin blue line" are casting police as combatants rather than public servants.
> Police don’t serve the community
Whom do they serve?
In my experience, mostly themselves.
The wealthy and powerful, in exchange for their preferential treatment, their place of power, and more generous scraps from the master's table.
Capital.
Within capitalist societies police protect the interests of the state, which itself exists to serve and benefit the capitalist class. Police "maintain order" to ensure the proletariat continues to submit to the state and participate in the capitalist machine, and they commit violence against anyone who does otherwise.
3 replies →
Police in your area maybe?
Look into the history and meaning of “thin blue line.” It’s an endorsement of the us vs. them mindset prevalent in American policing that is the source of many (most? all?) police scandals.
6 replies →
[flagged]
[flagged]
They are largely there to enforce the law and investigate transgressions of it. If you've ever reported a theft lately you'd know that they are not resourced - at least around here - to put any priority on protecting anyone's property. They have their hands full dealing with domestics and drug and alcohol related offending.
That seems like a fundamental misunderstanding of policing.
Maintainers seem more like HOA boards than police.
I like to think of us at the thin blinking cursor.
They couldn't have used a worst name if they tried.
why cannot more people learn about git and branching
you want your merges in? patch it in your own branch
the real crux of the issue is the quality of "the offical version", the real issue, then, is about official branding.
I remember back in the day there were multiple kernel versions by different mantainers.... then again I'm probably missing the forest for the trees or something
Because forked kernels is how we ended up with all those SBCs which could only run some ancient version of linux. If you can't mainline it, you are committing to maintaining a fork forever, and the more changes you require, the more maintenance that fork will be.
The Linux kernel quite intentionally makes maintaining a fork a pain in the ass to pressure vendors to mainline their drives. But now mainline maintainers are refusing to accept those drivers. So it's unlikely we will ever see Macbooks properly supported on Linux.
But the Rust advocates are already telling everyone that they will handle all maintenance of rust code in the kernel and existing maintainers won't have to do a thing. If that was true it wouldn't matter if the rust code is in a separate tree.
Upstreaming to linux-zen might work, not a kernel dev tho.
marcan addresses this argument in an adjacent thread:
> Out of tree doesn't work. People, both in the kernel and out of the kernel, want things in tree.
https://lore.kernel.org/lkml/1e8452ab-613a-4c85-adc0-0c4a293...
The kernel is continuously changing (which IMHO is also an issue), so keeping a fork requires a lot of work to continuously adapt to those changes.
This is why forks made for specific devices are seldom updated for more than a few months (or a couple of years if you're very lucky).
maybe we need something better than git. meh.
I feel like this problem should have a technical solution.
maybe a VCS good enough that this isn't as big an issue? (along the lines of pijul?) or
maybe something from formal verification methods to better enforce or more clearly explain why some drivers cannot be accepted. then again, I don't think the computer science is quite there yet
4 replies →
There are still a few custom kernels out there. Liquorix comes to mind.
https://liquorix.net/
not the phrase I'd choose to associate with, even metaphorically
This phrase has been largely co-opted by US white nationalist/fascist groups (similar to how Pepe the Frog was) but that is a relatively recent development; the phrase has been in use for a long time before that.
https://en.wikipedia.org/wiki/Thin_blue_line
it's not that recent that it's a very negative term - the wiki even says the usage with regards to covering up police crimes dates back to the 70s
2 replies →
[flagged]
unfortunately I did read it
Pre-supposes existing high quality, a fact not in evidence.
What are the higher quality alternatives of similar scope you are comparing against?
[flagged]
For real. I have a great deal of support for maintainers but that comparison is the quickest way for you to lose it.
Well tone police is finding themselves pretty hostile world lately.
[flagged]
[flagged]
[flagged]
I think that there's potentially a couple of things at play here:
Taken literally, the idea of a "thin blue line" protecting society from chaos has some authoritarian overtones.
It's a slogan that tends to be used to simply dismiss problems like police brutality. That is to say, it comes across less like a difference in policy position and more like indifference to the suffering of others.
The political faction that it's associated with, i.e. MAGA, is currently in the process of attempting to purge anyone with a worldview that they dislike from public society.
All of those factors creates a situation where it is difficult for those who use signifiers like "thin blue line" to practically operate simply as "people who happen to be conservative" as opposed to "people who represent an irreconcilable threat to people they disagree with".
Do you bring politics into the workplace when it is not relevant?
1 reply →
[flagged]
[flagged]
The burden of proof isn't on maintainers. Recognizing new, untried code invariably means bugs and instability to work through, often for questionable gains, is a mark of an experienced developer. Work your way through enough industry hype cycles and you'll see, new shit just causes new and usually weirder problems than whatever it was intended to replace.