Comment by Curiositry
5 days ago
This is great.
I frequently break my site in ways that aren't obvious. Right now, I use a combination of visualping and a homebrew tests.sh that hits various endpoints and runs checks, but I have been meaning to integrate screenshoting into my tests script (via selenium or cutycapt) rather than relying on a hosted service.
Have you found a good way of diffing the screenshots? DiffPDF works pretty well, but I haven't found a good solution for checking whether there are relevant changes automatically, rather than just has-changed, in a way that could be integrated into a script.
I only use the built-in diffing by Playwright. It comes with a nice overview page [0] that shows all the failed tests including traces and screenshots. There you have a pixel diff. If you have some notion of irrelevant changes that shouldn't warrant a test failure, I wouldn't know of a way to pull that off.
[0] https://playwright.dev/docs/trace-viewer-intro#opening-the-h...