Comment by bayindirh
10 months ago
[Putting my dusty Linux Distro Maintainer Hat on]
First of all, I wholeheartedly applaud Marcan for carrying the project this far. They, both as individuals and as a team proper, did great things. What I can say is a rest is well deserved at this point, because he really poured his soul into this and worn himself down.
On the other hand, I'll need to say something, however not in bad faith. He needs to stop fighting with the winds he can't control. Users gonna be users, and people gonna be people. Everyone won't be happy, never ever. Even you integrate from applications to silicon level, not everyone is happy what Apple has accomplished technically. Even though Linux is making the world go on, we have seen friction now and then (tipping my hat to another thing he just went through), so he need to improve his soft skills.
Make no mistake, I'm not making this comment from high above. I was extremely bad at it, and I was bullied online and offline for a decade, and it didn't help to be on the right side of the argument, either. So, I understand how it feels and how he's heartbroken and fuming right now, and rightly so. However, humans are not an exact science, and learning to work together with people with strong technical chops is a literal superpower.
I wish Hector a speedy recovery, a good rest and a bright future. I want to finish with the opening page of Joel Spolsky's "Joel on Software":
Technical problems are easy, people are hard.
Godspeed Hector. I'm waiting for your return.
100% agree.
For the last few years, I've been saying the following regularly (to friends, family and coworkers): communication is the hardest thing humans will ever do. Period.
Going to the moon, launching rockets, building that amazing app... the hardest thing of all is communicating with other people to get it done.
As a founder (for 40+ years and counting) I manage a lot of different type of people and communication failures are the largest common thread.
Humans have a very, very tough time assuming the point of view of another. That is the root of terrible communication, but assumptions are right up there as a big second.
On the Marcan thing... I just want to say, control what you can and forget the rest (yes, this is direct from stoicism). Users boldly asking for features and not being grateful? Just ignore them. Getting your ego wrapped up in these requests (because that's what it is, even if he doesn't want to admit it), is folly.
I contributed to Marcan for more than a year. I was sad to see the way it ended. I wish him well.
> Humans have a very, very tough time assuming the point of view of another. That is the root of terrible communication, but assumptions are right up there as a big second.
That's very true. I recommend some people to read "The Four Agreements", because that thin book has real potential to improve people's lives through active and passive communication.
Also worth being aware of Robert Kagen's adult development model [0] or something similar; that gives people a framework to go from "humans seem" to some actual percentages and capabilities.
Spoiler, but approximately 66% of the adult population make do without being able to maintain their own perspective independently of what their social circle tells them it is. I imagine that would make it extremely challenging to determine what someone else's perspective is. Especially if that perspective is being formed based on empiricism rather than social signalling.
And if we're making book recommendations, Non-Violent Communication is a gem of an idea.
[0] https://medium.com/@NataliMorad/how-to-be-an-adult-kegans-th...
2 replies →
> (yes, this is direct from stoicism)
stoics don't write multi-paragraph goodbye letters
What do you mean by this?
Marcus Aurelius wrote extensive personal reflections in his "Meditations". Seneca wrote detailed letters to friends and family discussing philosophy, life, and death. Epictetus discussed death extensively in his Discourses, but sure, they were philosophical teachings rather than personal goodbyes.
They focus on acceptance and equanimity rather than formal farewells.
That said, "control what you can and forget the rest" is indeed stoicism, albeit simplified.
Stoicism is a school of philosophy. There are many many words over thousands of years discussing the practices and virtues.
5 replies →
No, the other ones
> He needs to stop fighting with the winds he can't control. Users gonna be users, and people gonna be people. Everyone won't be happy, never ever.
Right - but it kinda sounds like he's facing headwinds in a lot of different directions.
Headwinds from Apple, who are indifferent to the project, stingy with documentation, and not inclined to reduce their own rate of change.
Headwinds from users, because of the stripped down experience.
Headwinds from the kernel team, who are in the unenviable situation of having to accept and maintain code they can't test for hardware they don't own; and who apparently have some sort of schism over rust support?
Be a heck of a lot easier if at least one of them was on your side.
> Headwinds from Apple, who are indifferent to the project, stingy with documentation, and not inclined to reduce their own rate of change.
That is part of the challenge he chose to take on.
> Headwinds from users, because of the stripped down experience.
Users can be ignored. How much you get users to you is your own choice.
> Headwinds from the kernel team, who are in the unenviable situation of having to accept and maintain code they can't test for hardware they don't own
You don't have to upstream. Again, it's not the kernel team that chose to add support for "hostile" hardware so don't try to make this their problem.
> and who apparently have some sort of schism over rust support?
Resistance when trying to push an entirely different language into an established project is entirely expected. The maintainers in question did not ask for people to add Rust to the kernel. They have no obligation to be welcoming to it.
> Be a heck of a lot easier if at least one of them was on your side.
Except for the users all the conflicts are the direct result from the choice of work. And the users are something you have to choose to listen to as well.
> The maintainers in question did not ask for people to add Rust to the kernel. They have no obligation to be welcoming to it.
Their boss, however, did ask for it, so yes, they do have an obligation to be welcoming to it.
5 replies →
Another uphill battle that I haven't seen anyone mention is just how good mobile AMD chips got a year or so after the M1 release. I wouldn't buy a Mac to run Linux on it when I can buy a Lenovo with equally soldered parts that'll work well with the OS I wanna run already.
A lot of it is simply AMD getting on newer TSMC nodes. Most of the Apple's efficiency head start is better process (they got exclusive access to 5nm at first).
8 replies →
And some of these Lenovos are relatively upgradable too. I'm using a ThinkPad I bought refurbished (with a 2 year warranty) and upgraded myself to 40 GB of RAM and 1TB of SSD (there's another slot too if I need it). It cost me $350 including the part upgrades.
3 replies →
The M1 reset expectations for laptop battery life and performance, and the result has been great for all platforms.
Well, nobody ever said it wouldn’t feel uphill both ways.
It's not just that "people are hard" - it was clear that this will end up this way the moment marcan started ranting on social media about having to send kernel patches via e-mails. Collaborating on software development is a social activity and stuff like convincing maintainers to trust you and your approach is just as important part of it (if not more important) as writing code. Not realizing that is a sure road to burnout (and yes, I'm just as guilty of that myself).
> Not realizing that is a sure road to burnout (and yes, I'm just as guilty of that myself).
Humans are shaped by experience. This is both a boon and a curse. I have been also been on the hot end of the stick and burned myself down, sometimes rightly, sometimes wrongly. Understanding that I don't want to go through this anymore was the point I started to change.
> Collaborating on software development is a social activity and stuff like convincing maintainers to trust you and your approach is just as important part of it (if not more important) as writing code.
Writing the code is at most 5% of software development IME. This is what I always say to people I work with. I absolutely love writing code, but there are so many and more important activities around that, I can't just ignore them and churn out code.
> Writing the code is at most 5% of software development IME.
This really depends on what you work on. And how good the managers are on your team. I talked to a manager at Google once about how he saw his job. He said he saw his entire job as getting all of that stuff out of the way of his team. His job was to handle the BS so his team could spend their time getting work done.
This has been my experience in small projects and in very well run projects. And in immature projects - where bugs are cheap and there’s no code review. In places like that, I’m programming more like 60% of the time. I love that.
But Linux will never be like that ever again. Each line of committed code matters too much, to too many people. Is has to be hard to commit bad code to Linux. And that means you’ve gotta do a lot of talking to justify your code.
I did some work at the IETF a few years ago. It’s just the same there - specs that seem right on day 1 take years to become standards. Look at http2. But then, when that work is done, we have a standard.
As the old saying goes, if you want to go fast, go alone. If you want to go far, go together. Personally I like going fast. But I respect the hell out of people who work on projects like Linux and chrome. They let us go far.
4 replies →
It really depends what kind of code and for which usage.
People might also live their hobby dev experience better if they were really coding for themselves without any expectation except pushing the code to a repo. As a hobby dev, you don't have to make package, you don't have to have an issue tracker, you don't have to accept external contributions, you don't have to support your users if you aren't willing to have this on your shoulder. You don't even need a public git repo, you could just put a release tarball when release is ready on your personal website.
1 reply →
5%? Sure there is a lot of activity around software. But out of week of 40 hours I most certainly code more than at most 2 hours. If this is your workplace I think it's dysfunctional.
10 replies →
This. So very much this. If you burn bridges and then need them later, yeah, things are going to be hard.
> it was clear that this will end up this way the moment marcan started ranting on social media about having to send kernel patches via e-mails. Collaborating on software development is a social activity and stuff like convincing maintainers to trust you and your approach is just as important part of it (if not more important) as writing code.
Yeah but FFS using email for patches when there are so much better ways of doing development with git? The Linux Foundation could selfhost a fucking GitLab instance and even in the event of GitLab going down the route of enshittification or closed-source they could reasonably take over the maintenance of a fork.
I get that the Linux folks want to stay on email to gatekeep themselves from, let's be clear, utter morons who spam on any Github PR/issue they can find. But at the same time it makes finding new people to replace those who will literally die out in the next decade or two so much harder.
It's an interesting phenomenon where people keep coming out of the woodwork and criticize the most successful software development project in history for doing it wrong.
They're not micro kernel! They're not TDD! They're not C++! They're not CVS! Not SVN! Not SCRUM! Not Gitlab!
Yet the project marches on, with a nebulous vision of doing a really useful kernel for everyone. Had they latched on any of the armchain expert criticism of how they're doing it wrong all these years we wouldn't be here.
2 replies →
> there are so much better ways
...which doesn't matter at all.
The people in charge decided on their preferred ways of communication. You may believe that there are better ways out there, and I may even agree with you, but ultimately it's completely irrelevant. People responsible decided that this is what works for them and, to be honest, they don't even owe you an explanation. You're being asked to collaborate in this specific way and if you're unable to do it, it's on you. If you want to change it, work your way to become a person who decides on this stuff in the project, or convince the people already responsible. Notice how neither of those are technical tasks and that they don't depend on technical superiority of your proposed methods either.
> Yeah but FFS using email for patches when there are so much better ways of doing development with git?
You are missing one point, namely that email is probably the only communication medium that's truly decentralized. I mean, on most email providers you can export your mailboxes and go to someone else. You can have a variety of email clients and ways to back up your mailboxes. No git clone, no specific mailbox or server is in any way special, I think Linus emphasized recently that they made efforts to ensure kernel.org itself is not special in any way.
Yes, I find Github's or Gitlab's UI, even with all enshittification by Microsoft and whatnot, better for doing code reviews than sight-reading patches in emails. And yet I cannot unsee a potential danger that choosing a service — any service! — to host kernel development would make it The Service, and make any migration way harder to do than what you have with email. Knowing life, I'd say pretty confidently that an outcome would be that there would be both mailing lists and The Service, both mandatory, with both sides grumbling about undue burdens.
Have you ever been in a project which had to migrate from, say, Atlassian's stack to Github, or from Github to Gitlab, or vice versa? Heck, from SourceForge + CVS/SVN to Github or similar? Those were usually grand endeavors for projects of medium size and up. Migrate all users, all issues, all PRs, all labels, test it all, and you still have to write code while it all is happening. Lots of back-and-forth about preserving some information which resists migration and deciding whether to just let it burn or spend time massaging it into a way the new system will accept it. Burnout pretty much guaranteed, even if everyone is cooperating and there is necessity.
But you could probably build tools on top of email to make your work more pleasant. The whippersnappers who like newer ways might like to run them.
I personally don't think GitHub's PR model is superior to e-mail based patch management for two reasons. First, e-mail needs no additional middleware at git level to process (I can get my mails and directly start working on my machine), plus e-mail is at least one of Git's native patch management mechanisms.
This is not about spam, server management or GitLab/Gitea/whatever issue. This is catering to most diverse work methods, and removing bottlenecks and failure points from the pipeline. GitLab is down, everybody is blocked. Your mail provider is failing? It'll be up in 5 minutes tops, or your disk is full probably, go handle it yourself.
So Occam's razor outlaws all the complex explanations for mail based patch management. The answer is concise in my head:
> Mailing list is a great archive, it's infinitely simpler and way more robust than a single server, and keeps things neatly decentralized, and as designed.
This is a wind we can't control, I for one, am not looking and kernel devs and say "What a bunch of laggard luddites. They still use e-mail for patch management". On the contrary, I applaud them for making this run for this many years, this smoothly. Also, is it something different what I'm used to? Great! I'll learn something new. It's always good to learn something new.
Because, at the end of the day, all complex systems evolve from much simpler ones, over time. The opposite is impossible.
6 replies →
You are missing the entire point. When you interact with a group of people who already have a culture and a set of practices/traditions, you have to play by their rules, build up credibility with that community... and then maybe, down the road, you can nudge them a little to make changes. But you have to have credibility first, have established that you understand what they do and understand why their preferences are the way they are.
If you approach it from the viewpoint that you have the solution and they are Luddites, you will influence no one and have no effect.
Marcan's career as a developer includes lots of development on hostile systems where he's jailbreaking various consoles to allow homebrew.
Asahi Linux is similar, given how hostile and undocumented Apple Silicon is, but it has a great amount of expectations of feature completeness and additional bureaucracy for code changes that really destroys the free-wheeling hacker spirit.
I understand. While I'm not as prolific as him, I've grown with systems which retrocomputing fans meticulously restore and use, so I had to do tons of free-wheeling peeking and poking.
What I found is being able have this "afterburner mode" alongside "advanced communications" capabilities gives the real edge in real life. So, this is why I wish he can build his soft skills.
These skills occupy different slots. You don't have to sacrifice one for the other.
Why not develop a distro based on BSD/darwin kernel then?
Probably a few reasons. For Darwin, there are a few small projects but I think they are all functionally dead. The benefit with Linux, or even the BSDs here is, sure you gotta port to the hardware, but you should get a good set of user land stuff running for 'free' after that. Lots of programs just need to be compiled to target arm64 and they will at the very minimum function a little bit. Then you can have package maintainers help improve that. I don't think any of the open source Darwin based projects got far enough to build anything in user land. So you'd probably just get the Darwin code from Apple, figure out how to build it and then build everything else on top of it.
The BSDs. You can fork a BSD. Maybe he could try to mainline into the BSD, but would probably face a similar battle with the BSDs. Right, one again, the benefit mainlining into linux, and there is some (maybe limited) support to include Rust, is you can narrow your scope. You don't need to worry as much about some things because they will just sorta work, I am thinking like upper layers of the kernel. You have a CPU scheduler and some subsystems that, may not be as optimized for the hardware, but at least it is something and you can focus on other things before coming around to the CPU scheduler. You can fork a BSD, but most would probably consider it a hard fork. I also don't think any of the BSDs have developers who are that interested in brining in Rust. Some people have mentioned it, but as far as I know, nothing is in the works to mainline any kind of Rust support in the BSD kernels. So he would probably meet similar resistance if he tried to work with FreeBSD. OpenBSD isn't really open to Rust at all.
4 replies →
> Asahi Linux is similar, given how hostile and undocumented Apple Silicon is, […]
«Undocumented» – yes, but «hostile» is an emotionally charged term that elicits a strong negative reaction; more significantly, though, it constitutes a flagrant misrepresentation of the veritable truth as stipulated within the resignation letter itself:
Which is consistent with marcan's multiple previous blog posts and comments on here. Porting Linux (as well as NetBSD, OpenBSD) onto Apple Silicon has been no different from porting Linux/*BSD onto SPARC, MIPS, HP-PA and other platforms.
Also, if you had a chance to reverse-engineer a closed source system, you would have known that «hostile» has a very specific meaning in such a context as it refers to a system that has been designed to resist the reverse-engineering attempts. No such resistance has been observed on the Apple Silion computing contraptions.
> No such resistance has been observed on the Apple Silion computing contraptions.
I think they even left a "direct boot from image" (or something similar) mode as a small door to allow Asahi Linux development, if not to accelerate a little bit without affecting their own roadmap. Even Hector tweeted about it himself!
I also think calling it hostile is a little far. I recall Hector making comments of, "yea, even though is not greatly documented, it does things quiet a few things the way I would expect" and I believe even applauded Apple on a few thing. I wanna recall it was specifically around the booting.
It's simple statistics. With a large enough sample size, you're going to always have a few very loud outliers.
Yeah, I want to give them accolades for the great work they did.
I just wanted to also add that users will be users. Once its out, there will be endless posts about "why X" and "why not Y". No matter what you do, lots of people are going to be displeased. Its just the way things go. I hope he will want to pick it up again after some time.