Comment by tptacek
2 years ago
Just taken on its merits, I think a case can be made that this is one of the most overrated pieces of technical writing of the last 25 years. What's true in it isn't interesting ("the importance of having users", "release early release often") and what's interesting isn't true ("Linus's law" being perhaps the most notorious example). Much of the insight is taken directly from Brooks. The whole piece has as its backdrop the development of Fetchmail, which is not a well-regarded piece of software.
What's notable about Cathedral is its timing; it did capture the zeitgeist of what was an important moment in the computing field, the moment where we transitioned from 386bsd-style hobby projects to an industry run on free and open source software. But Raymond isn't the reason why any of that happened, and much of his description of that moment is faulty; the rest of it is just a retrospective of the engineering decisions involved in the writing of a midlist mail processing utility (fetchmailrc syntax, password encryption, the now-largely-irrelevant distinctions between MDAs and MTAs).
Even the high-level organizing notion of "cathedrals" and "bazaars", which should have been a lay-up, hasn't really proven out.
> What's true in it isn't interesting (... "release early release often")
This feels like 20/20 hindsight to me given that "release early release often" was definitely not the "axiom" that we think of today, and one could argue the contrary opinion was more widespread in 1997.
I can easily remember some debates at a large consumer web company in the mid 00s, where at the time we released once per quarter. We were struggling with product quality, and a large, significant portion of the software dev team argued that the way to improve our product quality was to reduce the number of our releases to something like only 2 per year. The argument was that we needed more QA time, needed more time to run integration tests, needed more time to ensure changes from all the different teams wouldn't break each other before releasing.
In our current world of CI/CD, a web company releasing only twice a year sounds absolutely insane. Fortunately the loud "slow down our releases" contingent lost (which is evident simply by the fact that this large tech company still exists - if they had reduced their release cycles as some desired I firmly believe the company would have died long ago). But I just bring this up because the mantra of "release early release often" was definitely not blatantly self-evident in 1997, and I'd argue it only became "axiomatic" after tooling support for CI/CD got much better.
"Release early release often" was a mantra of "Extreme Programming", a closed-source commercial software development methodology that predates this article by about 4 years, and was au courant at the time Raymond was was writing. One of my big thematic criticisms of Raymond's article is that it doesn't seem especially in touch with how closed-source development worked at the time.
I'm sympathetic to the idea that C&B is overrated, but it was published in 1997, and XP was only being fleshed out at the C3 program in 1996. The Agile manifesto -- in 2001 -- was what gave it its major visibility bump.
Release early and often was in the air in the late nineties, but C&B was legitimately the first time many people got to hear about it.
4 replies →
> a closed-source commercial software development methodology that predates this article by about 4 years, and was au courant at the time Raymond was was writing.
I think you are misremembering. Check out the Wikipedia article on Extreme Programming [1] - The book Extreme Programming Explained wasn't published until 1999. Kent Back didn't even start working on the idea until 1996.
In any case, I'm not arguing that the idea of "release early and release often" was complete unheard of, but I am arguing that it was definitely not the standard and I would say the idea had more detractors than adherents at that time. There was still very much a battle between "waterfall processes" and "agile processes" that was really just starting to get going in the late 90s.
1. https://en.wikipedia.org/wiki/Extreme_programming
1 reply →
Is Extreme Programming a closed source commercial software development methodology? What makes it so?
I went through a period of my career where I dived head long into it, read Kent Beck's book, liked what I read. Tried pair programming, TDD etc, loved it. Found team that felt the same and had a great couple of years.
Given the book, the many conference talks etc and comparing it to other flavours of agile that went full corporate (Scrum, SAFE). I'm surprised to hear it described as closed source.
2 replies →
I personally agree with the frequent release crowd, but my company sells a web-based SaaS product and releases quarterly, my last company released a web SaaS product 2x a year and SalesForce famously releases closer to once per year, so a large part my the world is "still insane" based on your opinion.
I was introduced to Salesforce in the early 2000s by a user very happy that useful features showed up every quarter without an upgrade process, so they have regressed substantially if this is the case.
I think it's safe to say in general that a large part of the world is still insane and will always be insane. Perhaps it's just part of the human condition.
> a large consumer web company in the mid 00s
This sounds like Yahoo!
> what's interesting isn't true ("Linus's law" being perhaps the most notorious example)
> Even the high-level organizing notion of "cathedrals" and "bazaars", which should have been a lay-up, hasn't really proven out.
you and i live oceans apart. a year of working around NixOS and especially linux on mobile phones has me working in very bazaar-like systems — where every component is swappable/composable — and seeing/using Linus’ law daily — in that people from out of the blue will patch things i’d been unable to figure out and when i hit edge cases that look familiar enough in any software i install i will debug it & send fixes upstream.
i’m well aware the above is niche. on the other hand i’m fairly confident that niche would be inaccessible to most of us who are otherwise participating in it were it not leveraging these two concepts.
We do. As a vulnerability researcher, my take on "Linus's law" (which, like "Gell-mann Amnesia" isn't the product of the famous person it's named for) --- "given enough eyeballs, all bugs are shallow" --- is that has been effectively refuted by the recurrent discovery of grave vulnerabilities that have hidden in plain sight, sometimes for over a decade, and with the success of commercial vulnerability research work in eradicating those vulnerabilities.
I don't think "bugs are shallow", or the "many eyeballs" section of this essay, are particularly talking about _discovering_ bugs.
The author seems to have had a worldview in which bugs don't really matter if people aren't coming across them, and in which the difficult part of dealing with a bug is either reproducing it or getting from a symptom to a cause.
If you were in a world where those two things are true then I think he's probably right that "many eyeballs" would help a great deal.
22 replies →
"My favorite part of the "many eyes" argument is how few bugs were found by the two eyes of Eric (the originator of the statement). All the many eyes are apparently attached to a lot of hands that type lots of words about many eyes, and never actually audit code." -Theo De Raadt
>> you and i live oceans apart.
> We do.
which, just to clarify, i didn’t mean as “one of us contradicts the other” but that “we experience different slices of life which are guided by different priorities”. i very much wouldn’t want to be a security researcher in the open source areas i occupy now. likewise i wouldn’t want to use the development practices i experience in community open source were i instead building a fighter jet.
looking at the bizarre overlap between “open source contributors who like {Nix,Haskell/FP,Rust}” and “people i know who work for defense contractors” though, i’m actually optimistic that our distance could shrink in time.
None of the significant security vulnerabilities that I can remember have been "deep" - subtle things that require extensive familiarity with the specific codebase and could never have been found by a drive-by contributor, which is the idea that ESR is refuting. Debian keygen bug? Dumb one-liner. Heartbleed? Dumb one-liner. Goto fail? Technically a logic bug, but a two-line thing that could be understood by reading that one file without any additional context.
I've seen cases where exploitation was complex, but even then, that tends to be because the full exploit is chaining together several bugs, each of which is individually simple and could be fixed by a drive-by contributor.
29 replies →
It seems possible that some better eyeballs (possibly automated ones) have been developed and deployed en masse.
I've known ESR and RMS (who he's made his career trying to destroy) for well over 35 years, so I can testify that ESR has always been insufferably narcissistic, arrogant, and smarmy, but 9/11 sent him over the deep end of racism, islamophobia and misogyny (google "Is the casting couch a fair trade?").
His software work is unremarkable (fetchmail is trivial and buggy, and CML2 was rejected by the Linux kernel developers), and he's made his career not by writing software but by attacking real hackers like Richard Stallman and trying to bring down their work, not by actually constructively developing any useful software himself. Attacking real hackers and trying to discourage people from using "free software" isn't "hacking".
He used to call himself "Eric the Flute", and he would go on and on ad nauseum about his beloved "Teenage Mutant Ninja Turtle NetNews Reader" and how it was so much better than every other netnews reader. But he never collaborated with anyone, and he never released it under any license. Much more Bizarre than Cathedral.
In terms of code, do you know if he has contributed substantially to any open-source projects? I just about the fetchmail/CML2 from wikipedia.
I guess ncurses sort of, but.. edit- Wow, THE Don Hopkins?
reposurgeon. It was what was ultimately used to transition gcc to git.
Just to be clear, ESR is still a tool: https://np.reddit.com/r/lisp/comments/1b1wbl/clos_is_an_ugly... http://esr.ibiblio.org/?p=8730
He's also mostly an ineffective, unlikable idiot, which we should all be thankful for because he could actually be somewhat dangerous if he weren't.
The main criticism (IMO) isn't really the work itself but how it's handed down as wisdom. It wasn't really cathedral==proprietary and bazaar==open source. In fact it was about cathedral vs. bazaar in open source software and that story is more complicated and comes down to "It depends." There are certainly open source projects that have more of a cathedral aspect whether because of benevolent dictator model or, more recently, a strong commercially-oriented foundation. But there are also areas that look more like a bazaar which have some advantages with respect to innovation but can be... messy.
ESR is on record stating that the original focus of 'cathederal' was on GCC and Emacs, the terminology was applicable to proprietary software and top-down corporate cultures.
Yeah I never felt proprietary was the cathedral, more like GNU was the target there.
Yeah, it's weird how this anti-GNU piece became interpreted as a pro-Open Source piece.
The main thing I learnt from ESR was that you can, actually, do worse than Crazy Uncle Stallman.
But we got "Everyone Loves Eric Raymond", which made me feel less like the odd one out!
2 replies →
At least GCC. But proprietary vs. open was the common understanding.
1 reply →
I found the whole ncurses drama tragic and sort of ridiculous because ESR behaved in a very non-bazaar manner. Held on to that thing like it was a precious family heirloom.
Really lowered my opinion of this book and ESR a lot. Not that it was ever particularly high.
https://invisible-island.net/ncurses/ncurses-license.html#ju... for the essence.
He also held onto, refused to share or collaborate, but bragged incessantly and inappropriately about his glorious "Teenage Mutant Ninja Turtles Netnews Reader", during the early 1980's when I knew him from east coast science fiction conventions and the arpanet, when he called himself "Eric the Flute".
He'd corner unsuspecting people at parties, hijack ongoing conversations, and insufferably yap on and on and on about it, to the exclusion of anything else, dissing all the other competing netnews readers. And he steadfastly refused to collaborate with anyone else or share it, and of course he never finished it either. Now he hates to even talk about it. If you run into him and want to make him shut up, ask him about it and see how awkward his response is!
https://web.archive.org/web/20050315094239/http://docs.freeb...
[...]
>-- A tangential diatribe on the unsuitability of fetchmail -------
>As to fetchmail: it is an abomination before God. If someone in the press ever paid for an audit of the source code, the result would refute the paper "The Cathedral and the Bazaar" to such an extent that it could damage the Open Source movement, which has pinned so much on the paper, in ill-considered haste.
>ESR has constantly maintained that fetchmail is "not an MTA", and he is right: it could be, but it's not.
>When mail is delivered to a POP3 maildrop, envelope information is destroyed. To combat this, you would need to tunnel the enveleope information in headers. Generally, sendmail does not support "X-Envelope-To:" because it exposes "Bcc:" recipients, since fetchmail-like programs generally _stupidly_ do not strip such headers before local re-injection of the email.
>Without this information, it can not recover the intended recipient of the email. The fetchmail program delivers this mail to "root".
>The program has another bug, even if you elect single message delivery (in order to ensure a "for <user@domain>" in the "Received:" timestamp line. The bug is that it assumes the machine from which the download is occurring is a valid MX for your domain. Many ISPs use one machine to do the virtual domain expansion, and another to do final deliver into ISP hosts POP3 maildrops. The net effect of this is that it attempts to use the "for <domain-account@isphost>" stamp, since it does not reverse-priority order "Received:" timestamp lines.
>Similarly, fetchmail fails to order headers in "confidence" order. This means that, given an email with a "valid" (MX matches in the "by <MX>" and a "for <user@domain>" exists) "Received:" timestamp line, a "To:", "Cc:", or "Bcc:" line, or an "X-Envelope-To:" line (which must be configured, and which is terrifically screwed up by qmail, requiring un-munging), fetchmail -- takes the first one it sees, not the most correct one.
>Using the "To:", "Cc:", or "Bcc:" lines ("data") to do the delivery buys a spammer the ability to relay mail, though the route it must take is rather circuitous. It also means that if the "Bcc:" was properly stripped before handing the RFC 822 message to an MTA, or if you are a list recipient, that data is useless for recovering envelope information. This means that root gets all mailing list mail from lists which do not do recipient rewriting (like the SVBUG list does), and root also gets all mail addressed to non-existant local users that was intended for particular local users (all SPAM and all mail that was requested but not sent specifically targetted to a particular user, via email header data).
>Unfortunately, ESR would not accept patches for the mistaken MX problem, nor for the preference order problem, nor for the tunneled envelope information stripping problem. He seemed to be too busy with speaking engagements, and has since declared fetchmail to be in "maintenance mode", in order to demonstrate a recognizable commercial software lifecycle for an Open Source project, to give business the warm fuzzies.
>-- End diatribe ------------------------------------------------
yup. its hard to describe what it was like back then. i was young and the idea that this hobby open source internet thing i had been interested in for a few years, had become mainstream, with Linus on the cover of Forbes, was mindblowing. i knew maybe 2-3 people in my entire life who had ever installed linux and that was only because of college. nobody else cared. but then Linus and ESR became a celebrity. dotcom bubble was in mainstream news and open source was part of the bubble. internet, and computers, were going mainstream.
So that is why i thought Cathedral and Bazaar was cool. Because i was young. didnt have access to Brooks book or others, physically, monetarily, but also it wasn't even written in my language. ESR's Cathedral was a textfile i could get off my dialup. and there just wasn't much else talking about what he was talking about.
Took me a while to realize. oh. hes just kind of a aggro dude writing extremist politics and ... this is like pseudo sociology. and nobody uses fetchmail.
edit - this is weird to think about but it's almost like... how they say 1991 was the "year punk broke", Nirvana and all this alt music / college rock was selling millions of copies and headlining festivals. these guys went from obscure regional bands who have to wash dishes for a living in between gigs, to international fame and millionaires almost overnight. im guessing there is a rock equivalent of Catb and i guess theres an equivalent discussion about how the music version of CatB wasnt really that insightful. but it sure was popular amongst the young folks, who saw the world change before their eyes.
I had a similar experience around 2008 when I was first getting into the type of stuff we talk about around here. My media literacy wasn't what it is now, so I didn't know just how far outside the mainstream some of it was, and I certainly didn't pick up on the underlying politics.
As a corporate low level programmer far away from the SV world, this book was rather eye opening to me. It might not have been the best take on the world or proven out in history, but reading it helped me down the path of open source and understanding the differences between what I was doing and what was going on elsewhere. I'm glad C&B was written and that I read it when I did.
I've noticed some interesting things about open source projects that I don't think ESR mentions:
1. Most contributions from volunteers can't be used as-is and require significant work to incorporate; code reviews take a great deal of work, and requests for revision are typically ignored.
2. Paying maintainers and developers is usually the most effective way to enable them to work on the project long-term.
3. In the absence of formal structure, projects can become cults of personality (hopefully benevolent dictators) or community (growing the community is more important than actually creating good software), possibly to the detriment of the project.
4. Companies love it when you build software for them for free, and are eager to exploit any open source software, e.g. by selling it as a service. But they hate it when their own employees work for free on things that the company doesn't care about. They also don't like paying to develop anything (such as open source software) which will be given away for free, perhaps to their competitors.
I think much of the reason the essay was popular was not so much the actual contents but that the metaphors captured the imagination. Software developers weren't simply toiling for the man, but could instead be building a beautiful bazaar. Whether or not that is true or not is debatable, but i think it captured the imagination more so than fsf's overly political freedom based rhetoric.
Most of the OSS software I really love is neither... more of a well tended zen garden.
Cathedrals have all kinds of issues, and bazaars either dissolve into endless feature creep (inevitably with terrible defaults) or an exercise in bikeshedding while critical issues go untouched because they aren't "fun".
The issue I've seen is OSS has rarely scaled to large projects, with the Linux kernel being the exception. Part of why desktop Linux never happened was people kept arguing over desktop environments and UI toolkits, and it led to an incomplete, glued-together product. There wasn't a holistic vision, leaving design arguments to go on for decades.
Another problem is software in the purest sense stopped being the hard part. Look at Twitter, but this applies to lot of products that got popular after ~2000. The community is its asset, a lot of effort goes into just keeping it running, and human intervention (moderation) is/was often needed to tend to the garden.
OSS can be a building block, but the challenges we see today are at a higher level.
on a side note: I would argue that Unix-on-the-desktop for non-nerds was basically always a pipe dream (excluding highly locked down things like a chromebook), and indeed the unix philosophy itself doomed it. Lots of tiny tools chained together doesn't really translate to a GUI paradigm.
2 replies →
I would say most of it is stuff like TeX... not so much OSS as freeware that the author did not care to commercialize.
Also the analogy sinks to the level of not even wrong which is clear to anyone with a basic education in European history.
While you are rating things, let me chime in and say I still use fetchmail today. For some reason, I prefer a local Dovecot over everything else that needs my laptop to be online to browse the INBOX.
IIRC ESR also pontificated about dead bits vs. live bits and pretty much predicted the demise of the technical book industry which was massive at the time.
Edit spelling.
> What's true in [CatB] isn't interesting ("the importance of having users", "release early release often") and what's interesting isn't true ("Linus's law" being perhaps the most notorious example).
At the time or now? You were there, and so might mean the former, but I suspect most of the discussion here means the latter. And as a work of philosophy that has been assimilated into the mainstream, CatB has to look like a mix of trivially true and blatantly false things—that’s what mainstream acceptance of philosophy looks like[1,2].
[1] https://slatestarcodex.com/2013/04/11/read-history-of-philos...
[2] https://slatestarcodex.com/2019/01/17/highlights-from-the-co...
so much hindsight bias, sure, everything you take for granted now is not interesting, and for the parts that you think are not true you have just as little evidence as he had in 1997
If anything’s true about Internet / hacker / open source culture at the time, it’s that the bar was lower for someone to have a following of devoted fetishists willing to spread the Good Word. People can call it 20/20 hindsight all they want, and some of it is, but looking at all these sacred cows for what they are through a modern lens is important. ESR was, and is, a tool. So much of what was written and highly regarded Back Then is just poorly written self-congratulatory nerdy fairytales with an arguable connection to reality.
[flagged]
rektide> "Edit: anyone care to speculate on why I'm so quickly at -2? I think there's perhaps some ideogical disagreement here but it's not clear what is so principally intolerable about a perspective against such persistent & vociferous negativity."
My speculation: you're reading a lot more heat into 'tptacek's comments than is there, and raising the temperature of the discussion unnecessarily. For example:
rektide> "My god man."
rektide> "I don't see how you can so callously disregard the most central message & imagery of this book. Yes, the individual ideas are ill spoken & poorly emphasized pieces of the key revelation"
I don't read 'tptacek as callously disregarding anything. Those examples are used to support esr's thesis, and to use them to test whether or not the thesis is supported is fair. And "callously disregarding" is pretty charged language.
rektide> "...and in your 9 comments here so far you have nothing but bile to spill about them perhaps rightly."
Again, "nothing but bile" is pretty charged and in my opinion an inaccurate description of his comments. The most arguably hyperbolic part I read from him is "I think a case can be made that this is one of the most overrated pieces of technical writing of the last 25 years, and even that is tempered by the preface "I think a case can be made that". There are places where he's pretty declarative in his language, but nothing I read is pejorative or an attack/
rektide> "But I cannot broke this hit campaign you are launching, when you are so persitently cantankerous & negative, without ever going near the actual inner truth that it did remarkably connect a generation with. I wish you could have some balance & some ability to see value, Thomas."
There are numerous places where 'tptacek acknowledges positive aspects of the piece:
tptacek> "What's notable about Cathedral is its timing; it did capture the zeitgeist of what was an important moment in the computing field"
And is actively engaged in furthering the discussion:
tptacek> "I appreciate getting called on this and forced to think more carefully about it. Thanks for the response!"
rektide> "The true-feeling parts of Raymond's article read to me like a document of, for lack of a better term, late 1990s programming thinking."
Again, pretty charged.
Anyway, that's what I think is likely the cause of the downvotes from reading your comment in the context of what 'tptacek has written. But of course, I'm not a mind reader.
(I'm just sharing my impressions because you asked and in the hope that you find them useful: I'm not likely to follow up in this thread because discussions of downvoting and tone are rarely useful. I question the utility of me even posting this response.)
Thanks. That was a lot of time spent; I appreciate your commitment to inquiry/discovery.
I still struggle with what I see as a Thomas fighting a war of technicalities to avoid the actual point, and I tried to leave significant rope to allow his annoyances. And I don't K ow how to walk back what I still feel like is an injustice that hurts the truth so badly, don't see how temperance can be found against this. But I get my post's polarization better and see how the room is so happy to stick to Thomas here.
1 reply →
grr: looks like I didn't proofread this enough, including misattributions. Attempted corrections:
----
rektide> "But I cannot broke this hit campaign you are launching, when you are so persitently cantankerous & negative, without ever going near the actual inner truth that it did remarkably connect a generation with. I wish you could have some balance & some ability to see value, Thomas."
Again, pretty charged.
There are numerous places where 'tptacek acknowledges positive aspects of the piece:
tptacek> "What's notable about Cathedral is its timing; it did capture the zeitgeist of what was an important moment in the computing field"
tptacek> "The true-feeling parts of Raymond's article read to me like a document of, for lack of a better term, late 1990s programming thinking."
And is actively engaged in furthering the discussion:
tptacek> "I appreciate getting called on this and forced to think more carefully about it. Thanks for the response!"
----