Comment by jpadkins

I have found that having a separate agent (session / instance) do the benchmarking and reporting the results back to looping optimizing agent is a clean way to prevent cheating. The benchmarking agent has no reason to cheat, its goal is to just to run benchmarks when tickled.

I also found this is really nice for quality evals. Have one agent with no context on how something is made do a quality review, with lots of detailed feedback. Then pass back the review notes to the implementor for feedback. It works a lot better than having an agent self-evaluate its own quality.