Comment by rtpg
2 days ago
now sometimes that's 4 hours, but I've had plenty of times where I'm "racing" people using LLMs and I basically get the coding done before them. Once I debugged an issue before the robot was done `ls`-ing the codebase!
The shape of the problem is super important in considering the results here
You have the upper hand with familiarity of the code base. Any "domain expert" also necessarily has a head start knowing which parts of a bespoke complex system need adjustment when making changes.
On the other hand, a highly skilled worker who just joined the team won't have any of that tribal knowledge. There is a significant lag time getting ramped up, no matter how intelligent they are due to sheer scale (and complexity doesn't help).
A general purpose model is more like the latter than the former. It would be interesting to compare how a model fine tuned on the specific shape of your code base and problem domain performs.
People usually talk about how they're better than LLMs in the domains they're experts and with known codebases.
What about all the other, large amounts of cases? Don't you ever face situations in which an LLM can greatly help (and outrace) you?
Yeah totally, for unknown codebases it can help kick you off in the right direction (though it can send you down a totally wrong path as well... projects with good docs tend to be ones where I've found LLMs be worse at their job on this ironically).
But well.. when working with coworkers on known projects it's a different story, right?
My stance is these tools are, of course, useful, but humans can most definitely be faster than the current iteration of these tools in a good number of tasks, and some form of debugging tasks are like that for me. The ones I've tried have been too prone to meandering and trying too many "top results on Google"-style fixes.
But hey maybe I'm just holding it wrong! Just seems like some of my coworkers are too