Comment by pizlonator
6 months ago
This isn't control for confounding factors.
For example: folks are more likely to rewrite stuff that is well-understood, and stuff that is well-understood is going to have shorter review times and lower rollback rate.
That gnarly horrid mess that only a few greybeards grok and has massive test coverage, a long tail of requirements enforced by tests and experience, and a culture of extreme rigor? Longer reviews, more rollbacks, and less likely to be rewritten.
The first chart does in fact a compelling reason to believe the effect is not that at all. If the "easy" code was predominantly being rewritten you would expect to % new memory unsafe code and % memory safety vulnerabilities to scale at different rates as the difficult to work on areas remained in C and kept causing new memory vulnerabilities.
Personal experience also provides a compelling reason, my experience is absolutely that people are more inclined to rewrite the things that are causing trouble in a new language.
It's not a blinded randomly controlled trial of course, it's observational data. You can't be completely sure there isn't a confounding factor that explains the data, but it seems far more likely than not that it is a real effect.
I would expect memory safety vulns to be dropping in most C/C++ projects due to better practices
This contradicts what Google has reported about their own code, which is that most vulnerabilities are in new code
2 replies →
This is basically the C++ treadmill for decades and I think people are starting to realise they were duped.
Bjarne's C++ promised that if you use this instead of C you won't have these problems. The problems persisted of course. Then it was well you need to use standard C++ 98 not that crap pre-standard C++ you've been doing, once you adopt C++ 98 the problems will subside. Then it's you need "modern" C++ 11, of course you've got problems, that's because you used C++ 98, use this "modern" C++ instead.
By around 2020 they started to say the "modern" C++ 11 wasn't up to it, you need to write "contemporary" C++ 20 or better.
What was it George W Bush told us? "Fool me once, shame on...shame on you. Fool me...you can't get fooled again".
You would be severely disappointed.
> That gnarly horrid mess that only a few greybeards grok and has massive test coverage, a long tail of requirements enforced by tests and experience, and a culture of extreme rigor? Longer reviews, more rollbacks, and less likely to be rewritten.
I'd say that this is likely the most likely to be rewritten actually, because high test coverage is a massive enabler in such a rewrite, and because having a project that “only a few greybeards grok” sounds like a big organizational liability.
That being said, and while I'm pretty convinced that Rust bring massive benefits, I agree with you that these measurements shouldn't be taken as if it was a rigorous scientific proof. It's more of one additional anecdotal evidence that Rust is good.
> It's more of one additional anecdotal evidence that Rust is good.
But that means it's likely to be the worst kind of science:
- Group of people agree that Rust is good. This is a belief they hold.
- Same group of people feel the need to search for argument that their belief is good.
- The group does "science" like this.
And then the rest of us have a data point that we think we can trust, when in reality, it's just cherry picked data being used to convey an opinion.
> And then the rest of us have a data point that we think we can trust, when in reality, it's just cherry picked data being used to convey an opinion.
Calling what Google did here "science" and cherry picked is quite a disservice. It's observational data, but do you have any objection to the methodology they used? Or just (assumed?) bad vibes?
7 replies →
This article is mostly about new code.
This, from my limited experience working on AOSP at Google, only the newer code were being written in Rust, not the existing code. But that was 2 years ago.
That is somewhat mitigated by grouping comparisons of S/M/L change sizes.
It would be interesting to group changes by line-deletions, say to indicate rewrites (and size of them)