Comment by bena
9 months ago
That is quite a bit more thorough than I either expected or wanted. So, for that, many thanks.
Although in your excellent break down of the shortcomings of the book, you kind of make me want to go back and re-read it myself. Just to experience it from my current vantage point. To see where I disagree with the book now.
> Code Complete's table of "Average Cost of Fixing Defects Based on When They're Introduced and Detected"
This section really hits me because that's one thing I can confidently say I've internalized over the years. To know that it's been based on fictional data is disappointing.
You're welcome!
You, and I, and the entire software industry, it turns out.
https://buttondown.com/hillelwayne/archive/i-ing-hate-scienc... is the best summary I've found on that.
> Are Late-Stage Bugs More Expensive?
> (...) While there's no smoking gun, I think the body of research so far tentatively points in that direction, depending on how you interpret "late-stage", "bugs", and "more expensive". This is a newsletter, not a research paper, so I'll keep it all handwavey. Here's the rough approach I took to reach that conclusion:
> Some bugs are more expensive than others. You can sort of imagine it being a Gaussian, or maybe a power law: most bugs are relatively cheap, a few are relatively expensive. We'd mine existing projects to create bug classifications, or we'd interview software developers to learn their experiences. Dewayne Perry did one of these analyses and found the bugs that took longest to fix (6 or more days) were things like feature interaction bugs and unacceptable global performance, in general stuff that's easier to catch in requirements and software modeling than in implementation.
> I've checked a few other papers and think I'm tentatively confident in this line of reasoning: certain bugs take more time to fix (and cause more damage) than others, and said bugs tend to be issues in the design.
As I see it, this is not so much about how long the bug has been in the system but about what kind of bug it is. Unacceptable global performance or fundamentally incompatible requirements often kind of require you to restart your system from scratch, which means throwing away a lot of your code. But that doesn't mean that it's trivial to detect them (necessary to fix them) before you've written a lot of code, and throwing away all your code and design documents doesn't mean you're actually starting from scratch, because programming is theory building, not building construction. If writing "the same program" a second time takes you 30% of the time it took you to write it the first time (but the right way), in a sense you retained 70% of the work you did from the first time, even if not a line of code is the same. (Of course, a "bug" that costs 30% of the total project budget is a pretty big bug...)
Also, with iterative development, a requirements or design bug can be introduced very late in the project. Emacs has been around for 40 years but only got native-code compilation for Elisp (a huge requirements and design change containing, probably, many requirements and design bugs) in the last 6 years. 85% of Emacs development was before that. And people were testing it within months of embarking on adding it.
> As I see it, this is not so much about how long the bug has been in the system but about what kind of bug it is.
That makes sense. Even as I was working out in my head why a bug would be more difficult to fix later, I came across the same general thought. That the problem is that by the time the bug gets to production, there's a lot of underlying support for it. And often, fixing that bug means fixing the underlying support as well.
And it's something you often don't discover until you're bitten by it.
And these bugs will naturally be found later because they are more difficult to find in the first place.
Then we get to the bit that essentially reinforces the adage from The Mythical Man-Month: Plan to throw one away, you will anyhow.
And it makes sense. It is obvious when laid out as such. And it is obvious how that section of Code Complete is at odds with The Mythical Man-Month.