Comment by samdoesnothing
6 months ago
They found a memory safety bug in their Rust code and assumed it was the only memory safety bug in their Rust codebase. And then they compared it to the historical average in C++ code that's been around for almost two decades in production. I can't be the only one here who sees how biased this comparison is right?
Rather, they found one memory safety bug in their Rust codebase, and measured it against the legions of memory safety bugs they found in their C++ codebase. In neither case are they measuring against bugs not found, so no, it's not biased.
Except it's not an apples-to-apples comparison. The C++ code has been around a lot longer, and a lot of it was written with older versions of C++ which didn't have modern safety features. I'm sure there is a bunch of new/delete in their codebase still. And I'm sure they're actively looking for memory safety issues in C++, and probably not so hard (if at all) with Rust.
> The C++ code has been around a lot longer
They made an earlier report where they found out that older C/C++ code has actually a lot less new vulnerabilities compared to new code, so I guess here they are comparing to new C/C++ code to get the higher ratio, meaning the comparison should actually be apples-to-apples.
Is it a usual thing you do that when you're given data about a literal thousandfold improvement, in a context where there are well-understood theoretical and practical reasons why you might have expected to see such an improvement, you make up reasons why it is actually not an improvement at all, without either investigating to see whether those reasons are actually true or demonstrating that even if they were true, they could possibly explain more than a tiny fraction of the improvement?
2 replies →
You're sure of a lot of things.
3 replies →
If you want something more comparable, they estimate that only 5% of their Rust code is within unsafe blocks. That makes only 5% of the code with potential for memory safety issues: that's already a 20x improvement. Let's make it 10x because unsafe blocks tend to be trickier, but you still get a lot of Rust guarantees.
The thing is with Rust, you know where to look for memory safety issues: the unsafe blocks. C and C++? GLHF that's your whole codebase. As they mentioned, you don't opt-out of all of Rust guarantees by going the unsafe route. Of course you can ditch them, but that'll be hugely visible during code review. Overall, you can be much more confident saying "yup there's no bug there" in Rust than in C or C++.
Even if they are off by a factor of 100 it's a huge win and the point stands.
It's fair to point this out and worth the mention. Still, I'd like to think that the engineers behind this can at least gauge the benefit of this endeavor with some accuracy despite the discrepancy in available data, and stating the data that is available only makes sense.
and they found 1000 memory safety bugs and assumed it was the only 1000 memory safety bugs in that code which have been in production for 2 decades. How naive can they be?
Even so, relatively speaking if C is not 50,000x worse, it must be at least 2000x worse.