Comment by hathawsh
6 hours ago
The LLMs have been trained on all kinds of code. They often choose a hacky solution because that's what they encountered in similar situations in their training. It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
It generally only takes a few words to guide them to a better design than what they saw in the wild, but if you don't guide them, they follow the herd.
That certainly can be true but I would challenge "generally". If you're working on something novel then it's not exactly unusual for even SOTA models to produce a first attempt that makes some very bad assumptions but then get stuck when instructed to use correct axioms instead. This looks a lot like the suduko meme - without any real insight into the problem the models end up just iterating around in circles and never finding a solution.
You probably won't see this very often if you're working on simple CRUD web applications where there are millions if not billions of lines of code in the training data set. IME you will probably see it quite frequently if you're working on - for example - software to control some kind of equipment or analyse the results of some kind of experiment where the LLM might have little if any training data about the field you're operating in.
Any time the LLM can establish a tight feedback loop it can iterate effectively enough to make good progress. But if for one reason or another that feedback loop is long, the more guidance and direction they seem to need. If they cannot correct their course based on feedback from their own output they need an outside observer to push them towards different paths. CRUD stuff is the obvious easy answer. It represents some absurd amount of software development work and is drastically overrepresented in training data. For good and bad. Lately I've been working on a diffusion model for terrain generation. The goal is to be able to recreate realistic terrain based on water flow and erosion on top of plate tectonics for much longer geological time terrain adjustments. These are much longer feedback loops. Sometimes it'll spend 6+ hours training a diffusion model on my 5090 just for the quality to degrade compared to the last experiment. So letting the model go wild trying different approaches and evaluating the results doesn't work nearly as well as when it can iterate 1000 times in those 6 hours while checking its own answers.
That's a great perspective. Come to think of it, I wonder about the quality of the build123d code it generated when I vibe-coded a prototype complex assembly. Now I'm curious.