Comment by eru
16 hours ago
> The simplest practical property-based tests are where you serialize some randomly generated data of a particular shape to JSON, then deserialize it, and ensure that the output is the same.
Testing that f(g(x)) == x for all x and some f and g that are supposed to be inverses of each other is a good test, but it's probably not the simplest.
The absolute simplest I can think of is just running your functionality on some randomly generated input and seeing that it doesn't crash unexpectedly.
For things like sorting, testing against an oracle is great. But even when you don't have an oracle, there's lots of other possibilities:
* Test that sorting twice has the same effect as sorting once.
* Start with a known already in-order input like [1, 2, 3, ..., n]; shuffle it, and then check that your sorting algorithm re-creates the original.
* Check that the output of your sorting algorithm is in-order.
* Check that input and output of your sorting algorithm have the same elements in the same multiplicity. (If you don't already have a datastructure / algorithm that does this efficiently, you can probe it with more randomness: create a random input (say a list of numbers), pick a random number X, count how many times X appears in your list (via a linear scan); then check that you get the same count after sorting.
* Check that permuting your input doesn't make a difference.
* Etc.
Speaking for myself — those are definitely all simpler cases, but for me I never found them compelling enough (beyond the "it doesn't crash" property). For me, the simplest case that truly motivated PBT for me was roundtrip serialization. Now I use PBT quite a lot, and most of them are either serialization roundtrip or oracle/model-based tests.
Oh, yes, I was just listing simple examples. I wasn't trying to find a use case that's compelling enough to make you want to get started.
I got started out of curiosity, and because writing property based tests is a lot more fun than writing example based tests.
> The absolute simplest I can think of is just running your functionality on some randomly generated input and seeing that it doesn't crash unexpectedly.
For this use case, we've found it best to just use a fuzzer, and work off the tracebacks.
That being said, we have used hypothesis to test data validation and normalizing code to decent success. We use on a one-off basis, when starting something new or making a big change. We don't run these tests everyday.
Also, I don't like how hypothesis integrates much better with pytest than unittest.