Comment by beingflo
12 hours ago
Thanks for the example of a Playwright report page. I agree that getting browser tests (not even just visual tests) to work reliably is considerable work. I built out a suite at work for a rather complex web application and it certainly looks easier than it is. A couple of notes:
- I disagree that you need a powerful VPS to run these tests, we run our suite once a day at midnight instead of on every commit. You still get most of the benefit for much cheaper this way.
- We used BrowserStack initially but stopped due to flakiness. The key to getting a stable suite was to run tests against a local nginx image serving the web app and wiremock serving the API. This way you have short, predictable latency and can really isolate what you're trying to test.
> - I disagree that you need a powerful VPS to run these tests, we run our suite once a day at midnight instead of on every commit. You still get most of the benefit for much cheaper this way.
Then how do you know which commit is responsible for the regression? I can see that working for a very small team where the amount of changes is limited but even so, especially with css, where a change in some place can affect the styles in another.
We probably have max 50 commits a day in our team spread across many areas in the application. So when breakages occur it's typically easy to tell which commit caused it.
But I agree, if you have a large team or a large monorepo you probably want to know about breaking changes already at the PR stage.