Comment by wmanley
3 years ago
We used to use a similar approach for testing page objects in stb-tester. It was called stbt auto-selftest (it has since been removed). It would run your page objects against canned screenshots and generate a Python doctest capturing the behaviour of this page objects. You’d then commit the doctest to git and have ci set up to check correctness. See https://david.rothlis.net/pageobjects/characterisation-tests
This was really helpful - we could then see in or pull requests how changes to our page object code would affect the behaviour of that code - just look at the diffs.
It worked ok for a while, but we soon hit limitations with this approach. The big one is the interaction with git. We would get many merge conflicts in these files, particularly when rebasing to reorder commits. Sometimes we’d forget to regenerate the files and have to rerun and rebase after submitting to ci. It worked okish for us as we are sophisticated git users comfortable with rebase and filter-branch, and we’d designed the system so could debug issues effectively, but it wasn’t suitable for wider use.
We’ve moved away from this model to a slightly different one. We still generate this data[^1], but we store the data outside of git in a database indexed by git commit sha. We then display this data and associated diffs in a web-ui. We use GitHub checks to wire it back up to pull requests, so it’s just as visible as when it was committed to git, but more convenient and displayed in a nicer format. You can use the same web-ui to approve changes to this data.
[^1]: though in more structured form than doctests
There were other advantages to our new approach too: because all the running of the code is happening server side it is both easier and faster.
Easier because you don’t need to have a complete development environment available on the machine that you’re making the change. You can make a quick change in the GitHub web ui on mobile and see the results. This is particularly important in our case as we have some code that depends on specific hardware to run (CUDA), that isn’t just available on any developer machine.
Faster because the server can run the process in parallel across a farm of machines, which will be faster than running locally.
On a related note: I think that deterministic generating something based on a git commit and then later diffing the results across commits is in general a very useful technique. Our builds work similarly. We build our software into (several) container/disk images, then we show the diff between these build artefacts on PRs.
It can help you spot when something changes that you didn’t expect, and vice-versa.
For this to work we store all our builds indexed by git commit sha.