Comment by bob1029
5 hours ago
> If anyone can point an LLM at slow code and it automatically finds a hot loop and uses a trick it found somewhere on the 'net to vectorize it, there is little point in hiring someone with a focus on that.
LLMs are not very good at performance issues unless you walk into it with a clear idea of the likely root cause and the bigger picture regarding actual hardware and desired customer experiences.
"Please make the code go faster"
vs
"I am noticing what appears to be contention between threads under workload A, B & C, but not with workload D".
These are completely different universes of capability and outcomes.
Even if an LLM can fight its way to the answer on its own, you can achieve a specific desired result much faster and with significantly lower risk if you are genuinely an expert.
I've seen a concrete example of this recently. I profiled the client's product "the hard way" and arrived at a change to a single line of code that would eliminate a mutex issue. One of the client's developers used the LLM and wound up with a change set that touched hundreds of files, but otherwise achieved approximately the same performance fix. The other developer even had my hint that it was a single file change and couldn't figure out how to do this despite prompting a leading edge model regarding this exact possibility over and over.
Taste and aesthetics apply to absolutely everything. Not just UI/UX design. Perhaps it is even more important that we care about the things that are invisible to the customer. It is certainly easier to forget about them or treat them like they don't matter as much.
Not my experience at all. I pointed frontier models at some PyTorch code, told it to make it faster, and it optimized it 10x fold.
You’re probably limiting the frontier models by being specific.
I am sure people could find code such that an LLM can optimize it 10x fold. I have seen many types of code in my life (well written, horribly written, fast, slow, etc.).
I do feel there is a limit to LLM-s today. I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".
Knowing where that limit is, is as hard as it always was knowing how fast a team of engineers was going to do a project.
> I do feel there is a limit to LLM-s today.
I felt there were limits 4 years ago. I couldn't get even an 8k context window.
It's starting to feel like the important gaps are the only ones left that need to be closed before there aren't any left. It also feels like next year they will start meaningfully closing.
> I did not try, but I doubt it will work if I would tell it "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".
I do feel there is a limit to human-s today. I did not try, but I doubt it would work if I asked literally any programmer I know "make me a web browser that is bug free, perfectly secure, works as efficient as possible on my architecture and has best UX for me personally".
Hell, I'm willing to bet they'd fail at this task even with an unconstrained snacks and kombucha budget. Humans have a long way to go. My job is safe.
Not every optimisation is the same. LLMs can optimize code that is behind known practices, but won't usually invent new data structures or algorithms to solve a unique problem.
While it's unlikely for LLMs to implement patterns not present in their training, current training techniques absolutely can produce practices and data structures and algorithms that humans have never done before. Modern LLMs are not just trained on text. They are trained on computer use, including - at a high level - solving programming problems with code. They are not just rewarded for producing the most likely next token. They are rewarded for producing tokens that solve problems. Patterns and techniques can emerge from this that are not present in any human behavior.
1 reply →
This doesnt even apply to his issue. What does PyTorch code have to do with complex concurrency or complex systems in general.
Theres lots of bad Pytorch code, just go ask george hotz. This isnt the magic you think it is. And nobody wants to hire the guy that just points llms at things and says make it faster. Things have value because talented humans make them. Its why a luxury coat is worth more than the linens that make it, or one from walmart made by a machine. This will 1000% apply to programmers. I think OP will be fine.
The fact that you're pointing it to PyTorch code in the first place is being specific.
Pointing a frontier model to a 10 or 100MLoC codebase and saying "make this code fast, make no mistakes" doesn't work. As an experiment I recently tried this with a relative small (500KLoC) codebase and it got stuck on believing that the primary cause of slowdown was the database not using a connection pool. (Which was completely irrelevant for this specific code.)
With how you presented this, it makes it sound like it was either a toy codebase or something outside of your domain already.
How large is your codebase?
The example provided covered both options.
In general the OP is right, the people who get the most value out of LLMs are veterans.
I don't know why but in Typescript when I tell an LLM to write code to check if a character is of a type, it often uses a whole Set() instead of just using a string and doing indexof. It's weird because not only should strings be faster, doing it with strings should also be the prevalent way in the dataset. Yet it uses a Set.