Comment by hamdouni
3 years ago
I don't get it. In their example of fibo(15), they already wrote the fibo function but how do they know it is correct ? Ok it is nice to have an autofill for expected value but not from the same 'thing' that I want to test...
I'm not the OP but I think Fibonacci is a contrived example. In practice these "characterisation tests" are great for adding tests to an existing codebase. The "snapshot" just records the current behaviour; it doesn't judge whether it's correct or not, but if you refactor it will catch changes to the behaviour, and then it's up to a human to judge if the changes were desirable or not.
Fibo would be excellent for property testing.
I.e. For all x, Fib(x+2)=Fib(x)+Fib(x+1)
A property based test produces values of x and tests this property (or invariant).
If it fails the framework usually then tries to find the simplest failing case.
I am always slightly uncomfortable about property based tests that just replicate the logic of the thing they're testing. I don't really know of an alternative in this case, and obviously there's rarely any harm in having regression tests. But it feels like a code smell somehow.
4 replies →
Fib(x)=0 does satisfy this property though.
2 replies →
P.S. They're also great for writing tests for new code, for many of the reasons described in the OP.
I think what they've done is implemented a way of doing snapshot testing by putting the snapshot data in the test itself. In snapshot testing you just cache the result of a code block and use that cached data to check the code produces the same output as it did when you knew it worked. Putting the snapshot inline in the test makes it a lot easier to read.
IMO The real value in tests is making sure you don’t break things later, rather than for getting your current code working.
That’s a big assumption. In TDD you write the tests first, to help you achieve the results expected, this doesn’t fulfill that need. It is basically snapshot testing applied to any kind of function.
I am of the opinion that the true value of TDD is that tests get written at all. I suspect that in practice the tests tend to get written at roughly the same time as the code, rather than strictly before.
Maintaining tests that only test that things haven't changed can be a nightmare if you actually want things to change (like when you're making a change to the logic of a system). They can also be a real pain to maintain since if they fail you almost never know why they failed, just that they failed. This gives you (or the future maintainer) very little help in figuring out what's going on.
Guidelines for tests that reduce maintenance burden rather than add to it:
* Test the externally visible useful properties of your system, rather than implementation details. Related: don’t bother with mocks unless there’s no other way
* Keep your tests DRY: if there’s some invariant you want to check have it appear once in your test code
* Don’t over-test: you can delete tests and checks that fully overlap with other tests
* If you need to over-test make it easy to regenerate the expected results from your implementation (characterisation tests) as espoused by the article. * the amount of testing effort should be proportional to how important the functionality is. Reduce the effort for low impact features
IMHO, this is really more a documentation of the original behavior and maybe a test for consistency of later implementations against the original one. But it's not about validity. I'd call it a "traditionality test".
Not fib, but... I've written tests like this semi-recently for an algorithm that I had to devise. I went to Excel to generate the test data, based on a data pull from our production database. So, real production data, using a sequence of Excel spreadsheet functions, which also has multiple eyes on it (including legal, since this was regulatory in nature), fed as input to the algorithm to verify the output matched up.
I'm still on the fence about that but I'm not sure how else to test something that is essentially a glorified algebra formula.
Sometimes verifying a result is correct is easier than coming up with the result yourself.