Comment by jandrewrogers
5 days ago
The reaction to this article is interesting. I have found AI to be useful in software contexts that most people never exercise or expect based on their intuitions of what an LLM can do.
For me, a highly productive but boring use of LLMs for code is that they excel at providing midwit “best practice” solutions to common problems. They are better documentation than the documentation and can do a lot of leg work e.g. Linux syscall implementation details. My application domains tend to require more sophisticated solutions than an LLM can provide but they still save a lot of rote effort. A lot of software development exists almost entirely in the midwit zone.
Much more interesting, they are decent at reducing concepts in literature to code practice for which there are no code examples. Google and StackOverflow turn up nothing. For example, I’ve found them useful for generating specialized implementations of non-Euclidean computational geometry algorithms that don’t really exist in the wild that I’ve ever seen. This is a big win, it literally turns months of effort into hours of effort.
On the other hand, I do a lot of work with algorithms that don’t exist in literature, never mind public code, with extremely performance-engineered implementations. There is an important take away from this too: LLMs are hilariously bad at helping with this but so are human software developers if required to do the same thing with no context.
Knowledge for which there is little or no training data is currently a formidable moat, both for LLMs and humans.
I agree on the distillation step. They're very good at identifying what problem you're trying to solve, especially if you don't have much experience in the domain. I've found ChatGPT great for figuring out problems involving combinatorics. And then you can be canny and ask the LLM to write the next prompt to get some boilerplate code. I like to do this with ChatGPT 4.5 (heavily rate limited) targeting 4o.
Getting good code is still a pain with chat, but you can go a fair way by reducing the context as much as possible and only asking for core functions (no point wasting hundreds of tokens on an arg parser). I'd like to play with Cursor or Zed a bit more.
Debugging sometimes works. 4.5 often figures out errors in one, 4o often struggles and spits out suggestions like it's a customer support agent (it also falls into repetition fairly often if you tell it "no, so what else could it be?"). You really have to work to convince it up front that you're not an idiot.
The moat isn't intelligence, it's context
What? This is exactly what I would expect an LLM to be good at. Stack Overflow gives common solutions to common problems. LLMs give niche solutions to problems that aren't common enough for extensive discussion, but are still already solved if you find the right repository or study the documentation thoroughly.
Neither helps with the hardest and most time consuming problems faced by software engineers: solving problems that have never been (publicly) solved before, making hard decisions based on intuition about what will be necessary in the future, understanding the interactions between broad webs of dependency in large codebases, and working with proprietary domain knowledge.