Comment by wolvesechoes
7 hours ago
I always wonder what kind of projects are we talking about. I am currently writing a compiler and simulation engine for differential-algebraic equations. I tried few models, hoping they would help me, but they could not provide any help with small details nor with bigger building blocks.
I guess if you code stuff that had been coded a lot in public repos, it is fine, otherwise AI does not help in any way. Actually, I think I wasted more time trying to make it produce the output I wish for than it took me to do this myself.
That's been my experience. If it's been solved a million times, it's helpful. If you're out on the frontier where there's no public code, it's worse than useless.
If you're somewhere in between (where I am now) it's situationally useful for small sub-components but you need to filter it heavily or you'll end up wasting a day or two going down a wrong rabbit-hole either because you don't know the domain well enough to tell when it's bullshitting or going down a wrong path, or don't know the domain well enough to use the right keyword to get it to cough up something useful. I've found domain knowledge essential for deciding when it's doing something obviously wrong instead of saying "I don't know" or "This is the wrong approach to the problem".
For the correct self-contained class or block of code, it is much faster to specify the requirements and go through a round or two of refinement than it is to write it myself. For the wrong block of code it's a complete waste of time. I've experienced both in the last few days.
I don't even think you have to be on the frontier for LLMs to lose most of their effectiveness. Large legacy codebases with deeply ingrained tribal knowledge and loads of idiosyncrasies and inconsistencies will do the trick. Sad how most software projects end in this state.
Obviously LLMs in this situation will still be insanely helpful, but in the same way that Google searches or stack overflow is insanely helpful.
For me it's been toy games built on web languages, which happens to be something I toyed with via my actual raw skills for the past 15 years. LLMs have opened many new doors and options for what I can build because I now technically "know everything" in the world via LLMs. Stuff that I would get stuck wasting hours on is now solved in minutes. But then it ALWAYS reaches a point where the complexity the LLM has generated is too much and the model can no longer iterate on what it's built.