Comment by chriswarbo
3 months ago
> Just doing this as an afterthought by playing lottery and trying to come up with smart properties after the fact is not going to get you the best outcome.
This sounds backwards to me. How could you write any tests, or indeed implement any functions, if you don't know any relationships between the arguments/return-value, or the state before/after, or how it relates to other functions, etc.?
For the addition example, let's say the implementation includes a line like `if (x == 0) return y`; would you seriously suggest that somebody writing that line doesn't know a property like `0 + a == a`? Would that only be "an afterthought" when "trying to come up with smart properties after the fact"? On the contrary, I would say that property came first, and steered how the code was written.
Incidentally, that property would also catch your "always returns 0" counterexample.
I also don't buy your distinction that "real life code" makes things much harder. For example, here's another simple property:
delete(key); assert lookup(key) == []
This is pretty similar to the `a + (-a) == 0` example, but it applies to basically any database; from in-memory assoc-lists and HashMaps, all the way to high-performance, massively-engineered CloudScale™ distributed systems. Again, I struggle to imagine anybody implementing a `delete` function who doesn't know this property; indeed, I would say that's what deletion means. It's backwards to characterise such things as "com[ing] up with smart properties after the fact".
I agree with you. However the grand-parent comment has a point that it's not easy to extract testable properties from code that's already written.
It's much easier to proceed explicitly like you suggest: have some properties in mind, co-develop property tests and code.
Often when I try to solve a problem, I start with a few simple properties and let them guide my way to the solution. Like your deletion example. Or, I already know that the order of inputs shouldn't matter, or that adding more constraints to an optimiser shouldn't increase the maximum, etc.
And I can write down some of these properties before I have any clue about how to solve the problem in question.
---
However, writing properties even after the fact is a learnable skill. You just need to practice, similarly to any other skill.
> However the grand-parent comment has a point that it's not easy to extract testable properties from code that's already written.
True, property-based testing does not solve the problem of deriving the intended behavior of code where behavior is not documented (either by requirements documents, or code comments, or tests that aren't just examples but clearly indicate the general behavior they are confirming, or...)
OTOH, PBT can be used to rapidly test hypotheses about the behavior (though intent is another question) of legacy code, which you are going to need to develop and validate to turn it into code that is maintainable (or even to replace it with something new, if you need to generally be compatible.) Determining whether deviations from a hypothesized behavior are intentional or bugs is still an exercise for the user, though whether the deviations are highly general or narrow to specific cases can help to illuminate that decision, and a library like Hypothesis will help determine that.