Comment by metalspot
7 hours ago
I have done a fair amount of low level performance optimization with Opus 5 and its reasoning is still very poor. Like why is CRC so slow and going through loops until I ask it if is using hardware instructions and it tells me it is using its own hand coded implementation poor. Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
> Reasoning about l1/l2/l3 cache hit ratios and their implications basically throwing darts at the wall, in the wrong room. If you give it a benchmark feedback loop then it might get there eventually but still massive alpha for low level systems engineers who instinctively know how this stuff works and can now automate 99% of the grind.
I suspect a lot of the training set for this sort of thing is people online speculating about cache performance incorrectly.
Speaking as somebody who is a performance geek, my knowledge came from relentless experimenting over the years (starting in the 8 bit era). Beyond the basics, I haven’t seen much on high-performance engineering online. To learn, you need to do the hard yards and I think performance tweaking becomes almost instinctive rather than something driven by a hard set of rules.
At a low enough level, every performance tweak becomes unique and bespoke.
Of course, you could find people online talking about how to write high-performance code, but beyond a few basic techniques, their advice may not work for you — nobody can write a generalist article about performance engineering that will definitely solve the problem you have right now.
Arguably, there are fewer patterns for an LLM to infer as highly optimised code tends to become more and more opaque in the search for a nanosecond here or there.
I don't disagree, but IMO, a lot of code doesn't get to the point where those very low level techniques drive performance. Like, yes, if you are doing some heavy floating point math then that's where you end up needing it. However, in a lot of code finding hot paths and often simply switching out a O(n^2) for an O(n log n) or faster.
Getting and using tools to find hotpaths is generally the most important performance tweaking skill.
5 replies →
I would think it's that the kinds of places which value this kind of knowledge often have major disincentive to share it. I'm thinking of HFT firms as one example.
Around six years ago, I worked for a switch manufacturer and we had insanely optimized networking code that will never see the public light of day.
Can you give it the valgrind suite to loop over? I have yet to try that with AI, but maybe the cachegrind tool is enough to help it.
I see the same thing, except I was working on high level performance reasoning. Whenever a piece of code has multiple steps that require multiple algorithms to work, AI almost always fails to guess which step is the slowest and what causes that step to be slow. Even Fable makes wrong guesses. You definitely need to give them a benchmark feedback loop.
Are there actually any humans who can reason about things like cache performance from first principles? I know there are some people who think they can, and I suspect they're fooling themselves. The one iron principle of micro-optimization at the level of cache hits is "measure, measure, measure", you just cannot think your way to the right answer on the first try. Processors and instruction sets today are too complicated, and tips that worked on one generation might be neutral or worse on the very next revision, making all the cargo cult knowledge passed around on this topic at best useless. I'll echo one of the sibling commenters here and say that LLMs probably bullshit their way to answers on questions like this because that's what humans online do as well.
If you give an LLM a proper testing harness and feedback loop to actually generate hypotheses, test them and revise them, I suspect it will do much better.
Yes, people can demonstrably do this with high reliability.
Some humans carry detailed models of CPU microarchitectures in their heads, against which they can design code from first principles that will be nearly ideal on the first try. It is repeatable and verifiable. The best people can accurately predict the measured performance before writing a line of code.
Measurement is useful in cases where the model of software and hardware interaction is materially incomplete. In most cases this is because the people writing the software have insufficient understanding of the hardware. Having a limited understanding of the hardware is a choice.
It would be surprising if this wasn't possible.