Comment by __MatrixMan__

Agreed. I've got a crazy idea that I think might help...

Most tests have a step where you collect some data, and another step where you make assertions about that data. Normally, that data only ever lived in a variable, so it is not kept around for later analysis. All you get when you're viewing a failed test is logs with either exception or a failed assertion. It's not enough to tell a full story, and I think this contributes to the frustration you're talking about.

I've been playing with the idea that all of the data generation should happen first (since it's the slow part), it then gets added to a commit (overwriting data from the previous CI run) and then all of the assertions should run afterwards (this is typically very fast).

So when CI fails, you can pull the updated branch and either:

- rerun the assertions without bothering to regenerate the data (faster, and useful if the fix is changing an assertion)

- diff the new data against data from the previous run (often instructive about the nature of the breakage)

- regenerate the data and diff it against whatever caused CI to fail (useful for knowing that your change will indeed make CI happy once more)

Most tools are uncomfortable with using git to transfer state from the failed CI run to your local machine so you can just rerun the relevant parts locally, so there's some hackery involved, but when it works out it feels like a superpower.