Comment by fabianlindfors

Agreed, we had agents writing standard integration/unit/e2e tests but turned out to be a ton of work reviewing just that and hard to wrap one’s head around as a developer.

We ended up building our own test platform to reduce the human bottleneck in the verification loop.

Takes the form of a hermetic test environment that run all our code and infra in a way that matches production. Our tests interact with it from the ”outside”, e.g. by driving a browser, a terminal or making API calls.

We still have coding agents writing those tests but the big difference is that we record the browser and the API calls and can review and verify through a UI without having to do any manual testing. It also serves as a regression test suite with E2E coverage, which is kind of a must to avoid coding agents making assumptions and breaking things.

Not a huge fan of the term ”software factory” but working like this I actually think it’s achievable.