Comment by pimeys

What you want is a bunch of sessions to replay. Something anonymized if it's not yours, and something that's not depending on state.

You replay all your sessions against your harness, and then store all logs all output, everything to a safe place.

Finally use a blind judge to check everything, and score the output.

Then fix your harness, iterate again until better until you are in a point where it's just the model's weakness. If you get to that, use a bigger model.