Comment by zzzeek
2 hours ago
I deal with a lot of non-LLM users at work and quite often the conversation hits a dead end, where I'm like , ok but see, your app has this performance blockage due to your not having an index here - do some performance tests and you'll see (it was a hang caused by a FOR UPDATE locking the entire table due to lack of an index). And you can hear the pause (it's all over Slack) where they just aren't going there, because writing a performance suite for the issue in question would be a lot of effort to do by hand and in the "before times" would be a difficult undertaking to justify. Because they don't consider an LLM, a task they most certainly should be doing becomes a non starter. Never mind my own Claude had a whole plan ready to go to do this whole suite for them in about five minutes so they could study the impact of the index, but I really didn't want to just go ahead and do this all for them. At some point your non-LLM coworkers need to stare directly at work we'd never be willing to do before that's now trivial, and in fact is now part of the job.
Right, but performance testing has always been hard and LLMs are awful at it. So maybe your coworker was right?
I’m definitely on the “just use the clanker” end of the spectrum. But knowing when and is good. Constantly having to argue with people’s reposting of Claude’s idea of what’s wrong has left me burned out. At this point, if I have any idea about what’s going on and someone tells me “Claude says”, I will ignore them.
In this case, it sounds like you could have made a ten line repro that shows the problem. Why not just send that, something human interpretable?
What LLMs are you using? Just simply telling Sol 6.1, Opus 5.5, Fable or Astra to performance test something will get you a solid improvement in poorly optimized code.
Giving it a specific plan will get you a solid test harness.
And setting up an autoresearch system and running it overnight will get you expert level optimizations if you set the metric up right.
These things excel at performance optimizations.