Comment by Capricorn2481
6 hours ago
> If it can be measured, then LLMs can optimize it
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
6 hours ago
> If it can be measured, then LLMs can optimize it
Then they can start attempting to optimize it. They can also spin round and round making the numbers worse because they don't actually know what to do.
I had this experience at work trying to optimize a little high level Pytorch. It can't really get better than it already was, but LLMs were quite willing to pretend they will. The real solution is I need to open a PR for one of Pytorch's tracking issues.
Yeah, but that's just the scientific process of hypothesis -> evidence -> conclusion.
You need a measurement that can falsify hypotheses and reject branches that won't work.
Also, if all you have left in your project are performance issues that are hard to identify without flailing around (even with Fable/Astra) despite sampler/profiler reports, then you're doing really well and I wouldn't assume you're going to fare much better than the sota models in terms of stabs in the dark.
The point of this post is that this is explicitly not the case. If the metric is measured, the agent finds a way eventually (around 5 total tries typically unless it gets stuck), and learns from iterations where changes caused a regression after a revert.
In one case I used a made-up metric (since I didn't know the exact name or if it existed) and it somehow optimized that too.
> They can also spin round and round making the numbers worse
"Claude, if this idea doesn't measure as an improvement (use X benchmark and a T-test), discard it and try the next idea."