Comment by Espressosaurus
9 hours ago
That's been my experience. If it's been solved a million times, it's helpful. If you're out on the frontier where there's no public code, it's worse than useless.
If you're somewhere in between (where I am now) it's situationally useful for small sub-components but you need to filter it heavily or you'll end up wasting a day or two going down a wrong rabbit-hole either because you don't know the domain well enough to tell when it's bullshitting or going down a wrong path, or don't know the domain well enough to use the right keyword to get it to cough up something useful. I've found domain knowledge essential for deciding when it's doing something obviously wrong instead of saying "I don't know" or "This is the wrong approach to the problem".
For the correct self-contained class or block of code, it is much faster to specify the requirements and go through a round or two of refinement than it is to write it myself. For the wrong block of code it's a complete waste of time. I've experienced both in the last few days.
I don't even think you have to be on the frontier for LLMs to lose most of their effectiveness. Large legacy codebases with deeply ingrained tribal knowledge and loads of idiosyncrasies and inconsistencies will do the trick. Sad how most software projects end in this state.
Obviously LLMs in this situation will still be insanely helpful, but in the same way that Google searches or stack overflow is insanely helpful.