Comment by ActorNightly
6 months ago
The issue with most codebases is nobody thinks about starting out with acceptance testing system.
The way it should work is that before even writing code, you design a modular acceptance system that runs full suite of tests or a subset based on what you are working on.
This is essentially your contract for software. And on a modular level, it means that it scopes down the contracts to the individual sub systems. And things like memory and cpu runtime constraints are a part of this.
If you have this, you basically replace what the Rust compiler is doing for you with tests. Memory leaks are caught. However, as a benefit, you also can verify changes in the dev cycle with things like performance degradation, all in the same system.
I'm not sure that nobody thinks of this. We just have a finite amount of time. Usually with a solid approach, you get solid performance. Fixing a performance related bug rarely when it comes up, is still a time savings over designing this kind of rigorous process from scratch, and getting everyone on board with it.
Getting rid of a whole host of bugs due to the compiler is a big deal because you won't have to design this extra acceptance system or deal with keeping an entire organization disciplined by it. If you can solve this seamlessly I think that's an interesting product that others would be very interested in.
> because you won't have to design this extra acceptance system or deal with keeping an entire organization disciplined by it.
This is just a matter of tooling. (On that note, here is a free business idea, prompt engineer an LLM agent that sets this up for you)
While the compiler does do a lot of things for you, you still end up with things that you should check because compiler doesn't understand your program logic. If Rust was a absolutely strict typed language, where basically everything had a type that defined what data it could hold, then it would be a different story. For example, when parsing a json into an object, instead of strings/numbers/bools, every single field has a finite set of values that it can hold. Then the compiler can figure out a lot of things. For example, if you try to convert a string to int, if the string field type doesn't have a defined regex expression it must match, then the compiler can catch this.
Anything less then that, you are better of writing the validation system once and reusing it for all your code bases now and in the future.
> The way it should work is that before even writing code, you design a modular acceptance system that runs full suite of tests …
Sometimes. It depends on what you’re working on.
Part of the fun challenge in writing software is that the act of programming can teach you that you’re wrong at every level. The syntax can be wrong. The algorithm you’re implementing can be wrong. The way you’re designing a module can be misguided. And you might be solving the wrong problem entirely! Like, maybe you spend weeks adding a feature to a game and it makes the game less fun! Oops!
Tests formalise beliefs about what you want your code to do, at some level of abstraction. But if those beliefs turn out to be wrong, the tests themselves become a headwind when you try and refactor. You want those early refactoring to be as easy as possible while you’re learning a problem space.
Now, some programs don’t suffer from this as much. If you’re implementing a C compiler or drop in replacement for grep, you have some clear acceptance tests that will almost certainly not change through your project’s lifecycle.
But not all problems have clear constraints like that. Sometimes you’re inventing a programming language. Or writing a game. Or making a user interface. In my opinion, problems that are fully constrained from the start are some of the least interesting to work on. Where’s the discovery?
> Part of the fun challenge in writing software is that the act of programming can teach you that you’re wrong at every level. The syntax can be wrong. The algorithm you’re implementing can be wrong. The way you’re designing a module can be misguided.
I see you sir haven’t had experience with writing drivers for prerelease hardware. You’re right in these cases of course, you just aren’t right enough - the real fun starts when you can trust neither the hardware, nor the BIOS, nor the OS in addition to all the above.
I will simply say that Google is one of the few places that, to a first approximation, has this. We have our (pretty good) suite of tests. We can run only affected tests. We can them with instrumentation (*SAN), we can run them under various memory and CPU limits, we can run them 1000 times in parallel to deflake.
Anyway Google has all of that, and yet still finds this improvement.
Android testing standards aren't necessarily the same as the rest of Google. At least historically it wasn't, I'm sure things have improved.
Writing the tests before the code only really works if there's an interface that's fully defined and well specified in advance. There are definitely times where that is the case, but in my experience it usually doesn't work like that.
It doesn't work like that because people are afraid to change things up. Lots of time people want to see working code that works for nominal use cases before hand.
No it's because often designing the interface and writing the implementation go hand-in-hand.