Comment by pushpendraw
7 hours ago
one thing nobody mentioned here, once the agent is looping against the same benchmark it will happily optimize for the benchmark itself and not the real workload. worth rerunning the win against a slightly different input shape after, just to check it did not memorize the harness instead of actually fixing anything.
Maybe LLM's should be integrated with the software, so they could optimize to the real hardware and workload ?
There is an entire paragraph about benchmaxxing and how to mitigate that.