Comment by spion
3 years ago
Most of the non-joyful experience of writing tests lies in the context preparation, not so much in the expected outputs.
I think we need a modern version of design by contract instead. Its one of the most powerful techniques of testing software is unfortunately often ignored and overlooked these days...
Some less known things about design by contract:
- you can easily test way more than types in preconditions and postconditions
- extreme web controller postcondition example: every time there is a 200 response to a POST /new request, there should be at least one document more in storage than there was before the request. (caveat: unless there are deletes - then sum the deletes up too)
- things important to the business look a ton like contracts
- e.g. a diff of two legal documents always contains the full content (list of words) of both documents (never accidentally swallow content!)
- your postconditions can surprisingly often fully describe the desired function behavior
- with sufficient amount of pre and postconditions, your property tests look like "exercise the module or entire system with random actions" -> unit and integration tests
(see Hilel Wayne's excellent video on this topic https://www.youtube.com/watch?v=MYucYon2-lk)
Some things that could really help modernize design by contract
- Test at a certain sampling rate in production for expensive contracts (do not turn them off)
- Report contract violations via observability mechanisms (tracing, sentry etc)
- Better language support for contract DSLs
I would categorize this as part of integration testing, which I prefer above unit tests by light-years.
It can also be unit tests. You can take any function that has contracts defined and run a fuzzer (or a property testing generator) on its inputs, and its an isolated unit test.
Or you could generate input actions for the entire system to exercise all contracts, and get e2e tests.
Or you could run your code in production and exercise contracts (maybe at a certain sampling rate) to get observability and "testing in production".
Its the most powerful and flexible concept I know of, but it requires thinking in properties (pre/postconditions) which can be a bit tricky.
I agree, but unit tests are cheaper, you can do them unilaterally, and you can accomplish a lot with them in an organization that can't or won't invest in integration testing.
Even when integration tests exist, they typically don't go beyond exercising each ability of the system one or two ways. They don't achieve good "data coverage" or branch coverage, which is what property-based unit testing excels at.
There's an Elixir library that I found which fixed this aspect for me. assert_value let's you inline capture the result.
My overall happiness with testing improved significantly once I started using it.
https://github.com/assert-value/assert_value_elixir
> lies in the context preparation
The right fix here is to simplify the context preparation - make the functions more self contained and less dependent on external scope.
This is not always in your control - especially if you're using libraries and frameworks. But also there are sometimes complex functions that (need to) combine lots complex data (say from multiple APIs) together to produce results, and there is not much that can be done about it.
Take a structured document diff for example - its seemingly simple, diff(documentRepresentation1, documentRepresentation2) -> diffDocument. But in actual reality it has to handle all kinds of edge cases in the structure of the inner documents, and preparing the structure of those documents is hard enough that you actually need to build helper functions to make it easier to make a variety of them.
I don’t have sufficient experience to judge whether you are correct or not. But I hope that you are correct!
I think code deserves way more assertions and validations. And (like you alluded to) fine-grained ways to turn them on or off; we shouldn’t shy away from expensive tests that might take hundreds of milliseconds just because they might be non-practical to run everywhere—instead we should have configuration to turn them on or off. And not just simple on/off assertions like in Java but things that can have metadata like “cost”, “priority”, and so on.
And, of course, some things (probably the actual contracts) might be always-on.
There’s a lot of exciting potential!
Careful, you're coming dangerously close to reinventing SystemVerilog's formal verification support :) SV formal is tough at first but wonderful once you really get it because it lets you lay out your constraints and preconditions and then the solver verifies that your circuit fulfills those requirements.
I’m meaning to write a library that’s the intersection of design by contract and abstract algebra for Python. You’d be able to say that getting a diff is associative and get free tests just for documenting that.
another deep cut of testing is maintaining them, especially the blur between improper old test with changing specs, you this squared map to fix now.