Comment by mplewis9z
10 months ago
Marcan certainly can be abrasive (I mean lol, so can Linus), but all the things he points out in the message below are 100% valid - I highly recommend for anyone here to try to contribute something even very small and logical to the Linux kernel or git (which use similar processes), it’s an eye-opening experience that’s incredibly unapproachable, frustrating, and demoralizing.
https://lore.kernel.org/rust-for-linux/208e1fc3-cfc3-4a26-98...
Having read through the email thread, I think both vocal people are basically in the wrong here. There is a way to constructively disagree and the DMA maintainer did not do that. The Rust maintainer should not have brigaded using social media.
The “in hindsight” version of how this should have gone without ego:
* Patch adds Rust bindings to C API
* Maintainer has concerns about increased maintenance cost and clarifies policy about C changes and Rust abstraction if unsure.
* Since R4L is approved, the binding is allowed to exist in a form that doesn’t inhibit changes to C API. C API is allowed to break Rust (I assume, otherwise the entire effort is moot).
* Any concerns about tooling etc which DON’T exhibit this property (and can demonstrably show that merging this Rust code will make C code harder to change) are brought up.
* These are resolved as a tooling issue before the change is merged (I don’t think there are any in this case).
All the discussion about multi-language projects etc is for the project as a whole to decide, which happened when R4L was approved and the breakage policy was decided (might need to be properly formalised).
If the maintainer (or anyone) is unreasonable, then the only approach is to have someone with more authority weigh in and make the decision to bypass their objections or sustain them (which is sort of the direction this was going before the diatribes).
Both were wrong, but only one was corrected.
> If the maintainer (or anyone) is unreasonable, then the only approach is to have someone with more authority weigh in and make the decision to bypass their objections or sustain them (which is sort of the direction this was going before the diatribes).
While they were arguing, Linus said nothing. While the maintainer was issuing ultimatums, Linus said nothing. Linus only said something when social media forced his hand. This is the real issue.
You’re right - add insufficient leadership to the list as well.
IMO, it seems inconsistent to green light R4L and not declare a clear policy for Rust code interacting with C code without adding a hard dependency (and if it WAS declared, not enforcing it).
The only benefit of doubt I can give is that there wasn’t enough time for Linus etc to weigh in before the thread got sidetracked (and the decision became much more politically charged). It’s unclear what would have happened if only the maintainer was unreasonable.
Apparently GregH(?) had already stepped in earlier to resolve the issue before it blew up again. But I’ve not been following it closely.
>Both were wrong, but only one was corrected.
People are wrong in LKML often.
This time, somebody was wrong in a much worse way than usual.
While I never submitted a patch personally, I had once conferred with some of the input devs to add a trackpad to the synaptics driver... they were queueing up an update to add other trackpads, and they said they would add mine... 5 years later, it's still not there. It was just a one-liner, and I'm not really sure why it never got added...
On the other hand, I once ran into an issue with uboot where a bad update knocked out my emmc, usb and sata controllers... found an email address of someone developing the dtb files and got in touch with them, and it was fixed in under a week.
At the end of the day, people are weird sometimes. I wish all the best for marcan.
I tried once to contribute a fix to be able to use the track-pad on my laptop many years ago. But it was not accepted as the maintainer claimed it was an problem in userspace that did not process out of order events correctly. Despite none of the other drivers sent the events out of order. I had no intention to fix the problem on X11 (the only userspace for this at the time), so I used the patched kernel driver locally until I stopped using that laptop. https://bugzilla.kernel.org/show_bug.cgi?id=43591 https://lore.kernel.org/all/1340829375-4995-1-git-send-email...
FWIW, I have submitted a couple small patches to get the display and gamepad for my Lenovo Legion Go recognized correctly - probably similar levels of complexity to your change. One was to input and one was to display quirks.
They did take months to finally land, and the whole process of getting my corp email address to be compatible with their email-based PR system was way more of a faff than it had any right to be, but they did land. You can install mainline Linux on a Legion Go now and the display and controller will behave as expected, out-of-the-box.
Thanks!
> 5 years later, it's still not there. It was just a one-liner, and I'm not really sure why it never got added.
I think they expect people who want things to advocate harder than just mentioning it once. If no one brings it up again, then they assume that no one cares.
this seems very inefficient and the opposite of what I assumed. repeated requests take up time on both sides and are not a very good measure of how important something is.
7 replies →
why bother infering such intent when the obvious answer - that they simply forgot about it with no ill intentions - is right there?
1 reply →
Quite the opposite. If you “pester” for something, they’ll explicitly reject it.
2 replies →
People forget things etc.
Should probably have just asked again, or sent a small one-line patch. It's "mention something on Slack" vs "creating a GitHub issue/PR"
Then you get stories like Greg in the linked mail thread, who emailed to check after not hearing anything and got told that now he'd been annoying and it would never be done.
1 reply →
Which sounds inefficient and exactly the sort of problem that doesn't happen with a Github issue/PR.
6 replies →
Having contributed a few times, I'd rate it as similar (sometimes much easier!) than contributing to Firefox and Chromium. That is to say that it is indeed extremely time-consuming and frustrating, but when compared to projects of the same scale it does not necessarily come out as more time-consuming or more frustrating - this will never be a small team collaborating on a random Github repo. A simple "swap out X workflow for Y" does not fix these annoyances, and false dichotomies and peer pressure towards is not a way to cooperate.
I cannot claim to have felt the effects on the maintainer-side of this workflow in large-scale projects though.
It's way more painful to contribute to the kernel than contribute to Firefox, at least, unless things have changed since I was involved with Firefox.
Suppose you find a bug in the kernel and come up with a patch. You email the patch to some kernel mailing list and ask for feedback. Typically, you will receive no response whatsoever, because no-one is responsible for responding. You can try emailing random developers and eventually maybe one of them will have mercy on you.
In Firefox and I think Chromium, you can file a bug, attach your patch, request review from someone (the UI will help you choose a suitable person), and it's their job to respond.
In my experience it's the opposite - the email patch usually gets dealt with within a week or two, Firefox and Chromium dragged out because it wasn't whatever Mozilla or Google prioritized right now. Or worse, it might go against an internal corporate KPI.
In Firefox you have to fiddle with Mercurial, phabricator, and their homegrown CI. In Chromium its Gerrit and their homegrown CI, and oh btw you touched code that lacked tests so tag, you're it.
2 replies →
I haven't touched Gecko in a decade, but your second paragraph sounds like my experience. My best record was something like a single character bug fix taking months (might have been years?). Yes, the review flag was set to the right person.
I still remember the story where some other guys had to meet some Mozilla folks for lunch and nag them for reviews…
3 replies →
Besides the current drama, I'm glad someone of his stature agrees with and can call out the horrible processes and tooling involved in the kernel. Using email and a pile of hacks to mess around with patches just sounds nuts and makes it so much harder to understand or contribute. I don't think decentralized necessitates such a terrible workflow - you can run a local website with a distributed DB, distributed git forges exist, you can use federated chats instead of email, there has to be a better way than email.
I don’t think there is enough demonstrable benefit to sacrifice the ubiquity and flexibility of email for a newer solution, especially for existing maintainers who are probably very comfortable with the current workflow.
Harder to understand and contribute is a bad, but unless there is a proposal for a similarly flexible system that has minimal downsides and massive advantages, the preference of existing maintainers should dominate over potential future contributors. Especially factoring in how expensive of a migration it would be.
I can understand this mindset, but I also think this is how communities die. They go to great lengths to avoid inconveniencing existing members while neglecting to welcome new ones. In the short term, the better choice is always to favor the existing contributors right up until they start dropping out and there's no one left to replace them.
Linux is so ubiquitous & important that might never happen, maybe it will just become increasingly captured by corporate contributors who can at least build long lasting repos of knowledge and networks of apprenticeship to help onboard newbies. Open source in name only.
I really like the way sourcehut integrates mailing list patches with a good UI. I’d like to see that become more common in some of these “classic” open source projects.
Afaik Linus tried Github in the past, but had several significant complaints about it hiding information, messing with basic git operations, generating bad commit messages, etc. . So it is not as if they wouldn't use something better, there just isn't anything that has feature parity with a workflow they have been optimizing for decades.
That optimization includes things like email filters and email client customization that is individualized to longtime contributors, not to mention that it is just what Linus and others are used to. And the long time contributors have had years, or decades to incrementally set up their tools, and become familiar with the workflow. The problem is that new contributors and maintainers don't have that, and learning the workflow, and setting up tools so that the email based workflow is manageable is daunting and takes a lot of time.
I won't contest that there are advantages to the linux Kernel's workflow, but there are downsides too, and a major one is that it scares off potential contributors.
That said GitHub definitely is far from perfect as well, and has different strengths and weaknesses from email based flows. As do any other options.
But just because there isn't currently anything that is unilaterally better doesn't mean things can't be improved. There is clearly a problem with onboarding new developers to the linux workflow. That should be acknowledged, and a solution sought. That solution doesn't have to be switching to GitHub or similar. Maybe there just needs to be better documentation on how to set up the necessary tools, that is oriented towards developers used to the Github process. Maybe there needs to be better tooling. Maybe the mailing lists need to be organized better, or have the mailing list automatically add metadata in a standard, machine-readable format to emails. Etc.
> [..] a workflow they have been optimizing for decades.
it sounds the opposite of optimized to me. Unless we're optimizing for something other than developer experience and efficiency?
2 replies →
> a workflow they have been optimizing for decades
obligatory reminder about breaking someone's workflow https://xkcd.com/1172/
Every time I have to interact with mailing list based projects I feel like I must be missing some secret set of tools and workflows to make it easier.
Then I talk to the old timers and they act like I just need to get used to it.
I always thought it was a pretty blatant "vibe check" to filter out people who are so uncomfortable with software that they can't customize their environment to create an email workflow that works for them.
That sounds about right - the medium is the message. If you can't stand the clunky-but-working, decades-old, patch process, you probably won't stand the clunky-but-working decades-old code.
I'm grateful the kernel still supports MIPS, which means an old appliance of mine still works perfectly fine and is upgradable. I would be cery sad if someone were to rip-out support of an old MIPS arch, just because it's old and unwieldy
I've contributed to a couple of projects that use email based workflows. I can customize my environment, but it takes a lot of time, and I would rather do something else than figure out how to filter the firehose of a mailing list to the few emails I actually care about, or learn how to use a new email client that is slightly better and handling patches.
The first few times, it took me longer to figure out how to send a patch than it did to fix the bug I was writing a patch for.
8 replies →
Can't or won't? Surely what you just read would make you reconsider?
1 reply →
As someone who has never used mailing lists before (for software development), how much harder/less advantageous it is to migrate to an issues or thread-based approach, like with Github?
And why not?
The short version is:
- Distributing patches via email is more scalable than web hosting. Even GitHub could not host the level of activity happening on the LKML mailing list
- Web hosting has a variety of access and legal problems when working with international teams; email crosses borders more easily
- Email is a decentralized and distributed system that prevents any single company from controlling a project's development infrastructure (release infrastructure is another story, but multiple companies will generally manage their own release process anyway)
It's not Wikipedia, right? Getting the maximum number of contributors isn't a stated goal? I'm a C programmer with a fair bit of kernel experience, and they don't want me, I'm pretty sure, and I'm completely fine with that.
Wikipedia has plenty of gatekeeping too. I once had to submit a single edit three times before the moderators safeguarding the article begrudgingly accepted it.
They do, but they have a stated goal of maximizing contribution. Linux does not, right? I'm asking.
5 replies →
Different Wikipedia communities have different governance policies. In the math wikis there's generally a rule that small fixes are not allowed. This stops people from arguing whether slightly better explained sentences are the right edits.
> Marcan certainly can be abrasive (I mean lol, so can Linus)
My impression of a few glancing online interactions is that they're both abrasive but marcan is quite unwise in a way that Linus has had beaten out of him
I'm tired of anaphoras.
And he's not just abrasive He's a troublemaker. Seriously, code of conduct violation? It was perfectly clear what Hellwig meant by "cancer".
In my opinion, calling the well-intentioned hard work of others "cancer" is undeniably hyperbolic and dismissive. It is clear that Hellwig used it in this way. To interpret it differently requires bending the mind. Most people would also consider it rude, but I'll grant that rudeness is more subjective.
There is an argument that being hyperbolic, dismissive, and maybe a bit rude isn't as bad as some people make it out to be, and that it is a fine way to hash out disagreements and arrive at the best solution - that it is simply a result of the passion of exceptional engineers. There has historically been much of it in kernel development. But it seems that as the background and culture of kernel maintainers has broadened, that a more measured and positive approach to communication is more constructive.
It doesn't seem like marcan exemplifies this very well either. It is a loss for someone so skilled to abandon collaboration on the kernel, and seems like the unfortunate result of someone being dismissive and rude, and someone else taking that harder than is warranted or healthy.
"To interpret it differently requires bending the mind."
Stange, I think interpreting it your way requires bending the mind. Hellwig clearly used it to describe what he sees at the ill effects of multiple languages in the kernel. It was not used to describe either Rust the language or this specifically this particular submission.
7 replies →
Even if you put that aside, the problem is you offer Hellwig two solutions and he NACKs them both.
I'll bite and play devils advocate here - both of those are not a solution to his problem. Ultimately he's the maintainer and he gets the emails if X driver is broken, so because of that he doesn't want to rely on another group to maintain the 'Rust half' of his part of the code. It's also a system that works until it doesn't, the biggest rule of the kernel is no breaking userspace - at some point in the future it won't matter if it's his C changes breaking the Rust drivers, it's still his changes that have to be rolled back if the Rust code isn't updated.
And to clarify I'm not saying he's right or wrong or acting good or bad. I have however expected R4L to ultimately fall apart because of this exact issue, the maintainers have never been on board with maintaining Rust code and that hasn't changed. While that remains the case the project is going to be stuck at a wall - to the point that if they're confident they can maintain the Rust code themselves they should just fork it and do that. If it works well enough they'll eventually be too popular to ignore with people choosing to write their new modules in Rust instead.
3 replies →
What is the interpretation of "cancer" in this context that isn't rude, offensive, or hostile to the R4L project?
He meant to say "the Rust code will spread everywhere [like cancer]".
I agree it's rude, offensive, and hostile, but there are degrees of things and context matters. "You are cancer" would be much worse. I feel we should try and interpret things in good faith and maintain some perspective. For a single word like this: you can just read over it (which is also what the other Rust people did).
Certainly outright removing Hellwig from the Linux project, as Marcan suggested, is bizarrely draconian.
As I argued a few days ago: part of "being nice" is accepting that people aren't perfect and dealing with that – https://news.ycombinator.com/item?id=42940591
He wasnt talking about Rust specifically, he was referring to codebases in any other language.
He said: “ And I also do not want another maintainer. If you want to make Linux impossible to maintain due to a cross-language codebase do that in your driver so that you have to do it instead of spreading this cancer to core subsystems. (where this cancer explicitly is a cross-language codebase and not rust itself, just to escape the flameware brigade).”
Mixing codebases in the Linux core.
I think the conversation is more about people equating R4L as validation for rust or even themselves.
1 reply →
> It was perfectly clear what Hellwig meant by "cancer".
No, it is not perfectly clear.
The generous interpretation is that they meant it is "viral", spreading through dependencies across the codebase (which I don't think is technically accurate as long as CONFIG_RUST=n exists).
The less generous way to interpret it is "harmful". The later messages in the thread suggests that this is more likely.
But I'm left guessing here, and I'm willing to give some some benefit of doubt here.
That said, the response to this comment was incredibly bad as well.
And Linus’ immediate reply
https://lore.kernel.org/rust-for-linux/CAHk-=wi=ZmP2=TmHsFSU...
(not taking either side, just interesting to read the reply)
This part of the reply exemplifies one of the big problems in the kernel community:
> You think you know better. But the current process works.
Regardless of how badly broken the kernel development process is, Linus and others observe that Linux continues to dominate and conclude that therefore the development process "works". Success in the market blinds the successful vendor to their defects. Sound familiar?
If Linux carries on down this path then eventually something else that has been subject to more evolutionary pressure will displace Linux, and we'll all be telling stories about how Linux got complacent and how it was obvious to everyone they were heading for a fall.
And honestly, with maintainers like Hellwig maybe that's what needs to happen.
Or worse: Linux is so widespread and managed to practically kill most Unix alternatives, that progress in OS development is slowed down globally. I would strongly prefer Linux being an OS with a lot of progress to stagnation and possible no alternative in the next decades.
I find this reply interesting. Linus says that what matters is technical stuff, but even before the social media brigading, the whole thread was nothing but non-technical drama. So why is Linus focused only on that and not Hellwig's behavior?
You have to be pretty clueless not to understand that Martin's is wrong here, he, and the rest of Rust bozos he clicks with should have been kicked out of the Kernel the minute they started with their social media drama... of course, drama and rust are just bound to be hand in hand.
Definitely interesting to read both sides. I think they both present compelling arguments. There's a need to ensure stability with the kernel and avoid interference with outside forces. I suppose balancing that principle with eventual change is an inevitable difficulty.
Up until the point that he tried to leverage social media to get his way in a kernel maintainer dispute? That's just fundamentally not acceptable.
Linus was right to reprimand him for the suggestion.
I don't think there's "the point" when it was pretty much modus operandi for years.
I agree with you, I'm just not personally familiar with their behavior and was trying to be as charitable as possible.
I've contributed here and there over there years, even got something merged that broke Linus's printer driver. It really isn't unapproachable, frustrating, or demoralizing.
You broke Linus’ printer driver and you’re still alive to post? WOW! ;)
Looks like @marcan deleted his existence on mastodon? Does anyone have a copy of what he said on there?
The tweet he got called out for on the thread was
"Thinking of literally starting a Linux maintainer hall of shame. Not for public consumption, but to help new kernel contributors know what to expect.
Every experienced kernel submitter has this in their head, maybe it should be finally written down."
The person who called him out for that made some testy social media comments of her own this morning.
Personally, seeing
> Being toxic on the right side of an argument is still toxic, [...]
written unironically, on social media, immediately after that person wrote @marcan
> and if that then causes you to ragequit, because you can't actually deal with the heat you've been dishing out coming back around the corner: fuck off
leaves me feeling more sympathetic to marcan's argument about the kernel being full of toxic attitudes, not less. Maybe public shaming isn't the answer but there's a problem here. Maybe don't make comments like that on social media if you want to criticize people for leaning on social media in kernel disputes.
6 replies →
> Every experienced kernel submitter has this in their head, maybe it should be finally written down
If true, then it sounds like there are some “missing stairs”[1] (the professionally difficult kind, hopefully not the other kind) in Kernel development.
[1]: https://en.wikipedia.org/wiki/Missing_stair
You can use the waybackmachine:
https://web.archive.org/web/20250204162031/https://social.tr...
However it seem that you need to disable js as soon as the content load or it will be overwritten by a 404
As a follower of him on Mastodon this makes me sad. Hector posted a lot of valuable, very informative toots that I learned a lot from.
Haha it’s funny that this stuff is still going on. The difficulty of getting things into mainline is why Con Kolivas stopped developing his interactivity-prioritizing schedulers for Linux some 20 years ago. It’s just how the project works.
I agree contributing code to the kernel is by no means as approachable or easy going but it's not self-evident that alone is supposed to be the sign of bad things™ unlike more specific examples boiling up to be part of that picture. Are there things and ways I think it could be improved? For sure! I just don't necessarily think they imply the resulting process would be quick and painless.