Comment by masklinn
6 months ago
Most of these is confirmation of easily observable reality, but the 4x difference in rollback rates, jesus christ.
6 months ago
Most of these is confirmation of easily observable reality, but the 4x difference in rollback rates, jesus christ.
We all knew rust was safer. It was the 1000x number that surprised me.
Which is more that rust isn’t that safe in my mind, it’s that bugs are that prevalent. I never would have guessed that.
That 4x rate is very impressive too.
Great seeing all this data from a large big complicated codebase.
The issue with most codebases is nobody thinks about starting out with acceptance testing system.
The way it should work is that before even writing code, you design a modular acceptance system that runs full suite of tests or a subset based on what you are working on.
This is essentially your contract for software. And on a modular level, it means that it scopes down the contracts to the individual sub systems. And things like memory and cpu runtime constraints are a part of this.
If you have this, you basically replace what the Rust compiler is doing for you with tests. Memory leaks are caught. However, as a benefit, you also can verify changes in the dev cycle with things like performance degradation, all in the same system.
I'm not sure that nobody thinks of this. We just have a finite amount of time. Usually with a solid approach, you get solid performance. Fixing a performance related bug rarely when it comes up, is still a time savings over designing this kind of rigorous process from scratch, and getting everyone on board with it.
Getting rid of a whole host of bugs due to the compiler is a big deal because you won't have to design this extra acceptance system or deal with keeping an entire organization disciplined by it. If you can solve this seamlessly I think that's an interesting product that others would be very interested in.
> because you won't have to design this extra acceptance system or deal with keeping an entire organization disciplined by it.
This is just a matter of tooling. (On that note, here is a free business idea, prompt engineer an LLM agent that sets this up for you)
While the compiler does do a lot of things for you, you still end up with things that you should check because compiler doesn't understand your program logic. If Rust was a absolutely strict typed language, where basically everything had a type that defined what data it could hold, then it would be a different story. For example, when parsing a json into an object, instead of strings/numbers/bools, every single field has a finite set of values that it can hold. Then the compiler can figure out a lot of things. For example, if you try to convert a string to int, if the string field type doesn't have a defined regex expression it must match, then the compiler can catch this.
Anything less then that, you are better of writing the validation system once and reusing it for all your code bases now and in the future.
> The way it should work is that before even writing code, you design a modular acceptance system that runs full suite of tests …
Sometimes. It depends on what you’re working on.
Part of the fun challenge in writing software is that the act of programming can teach you that you’re wrong at every level. The syntax can be wrong. The algorithm you’re implementing can be wrong. The way you’re designing a module can be misguided. And you might be solving the wrong problem entirely! Like, maybe you spend weeks adding a feature to a game and it makes the game less fun! Oops!
Tests formalise beliefs about what you want your code to do, at some level of abstraction. But if those beliefs turn out to be wrong, the tests themselves become a headwind when you try and refactor. You want those early refactoring to be as easy as possible while you’re learning a problem space.
Now, some programs don’t suffer from this as much. If you’re implementing a C compiler or drop in replacement for grep, you have some clear acceptance tests that will almost certainly not change through your project’s lifecycle.
But not all problems have clear constraints like that. Sometimes you’re inventing a programming language. Or writing a game. Or making a user interface. In my opinion, problems that are fully constrained from the start are some of the least interesting to work on. Where’s the discovery?
> Part of the fun challenge in writing software is that the act of programming can teach you that you’re wrong at every level. The syntax can be wrong. The algorithm you’re implementing can be wrong. The way you’re designing a module can be misguided.
I see you sir haven’t had experience with writing drivers for prerelease hardware. You’re right in these cases of course, you just aren’t right enough - the real fun starts when you can trust neither the hardware, nor the BIOS, nor the OS in addition to all the above.
I will simply say that Google is one of the few places that, to a first approximation, has this. We have our (pretty good) suite of tests. We can run only affected tests. We can them with instrumentation (*SAN), we can run them under various memory and CPU limits, we can run them 1000 times in parallel to deflake.
Anyway Google has all of that, and yet still finds this improvement.
Android testing standards aren't necessarily the same as the rest of Google. At least historically it wasn't, I'm sure things have improved.
Writing the tests before the code only really works if there's an interface that's fully defined and well specified in advance. There are definitely times where that is the case, but in my experience it usually doesn't work like that.
It doesn't work like that because people are afraid to change things up. Lots of time people want to see working code that works for nominal use cases before hand.
1 reply →
If they use Rust for new code and C++ changes are all in old code, this could be explained just by older code being more risky to change.
Funny, another commenter on this post was saying the opposite, that Rust was likely being used to just port existing features and that was easier because there were probably good tests for it already.
If you've actually written considerable amounts of Rust and C++, these statistics don't require justification. In my opinion it's completely expected that Rust code is easier to write correctly.
As a relatively novice programmer who's worked in tech for decades but not as a software developer: I take issue with the idea that you need to write considerable amounts of Rust and C++ for these statistics to be expected. In fact, despite Rust's initial vertical learning curve I'd say that any junior developer trying to implement anything with any degree of complexity at all in Rust and C++ would see the benefits.
At the very least, the fact that IDE integration can tell you all kinds of stuff about what you're doing/doing wrong and why accelerates things greatly when you're starting out.
4 replies →
I’d say the same applies for Swift vs ObjC.
Let’s end the C era.
5 replies →
I think they're comparing new code in Rust vs new code in C++.
I found it interesting that the rollback rate remained more or less constant despite size differences.