Comment by CjHuber

The problem is not in making something you're unfamiliar with. The problem is doing something that your familiar with, trying out an LLM to see if it can assist you, then you are kind of impressed for the first few prompts so you let it off the leash and suddently you find yourself in a convoluted codebase you would never write that way with so many weird often nonsensical things different to how you normally approach them (or any sane person would) so that you can basically throw it all in the trash. The only way this can be avoided is by diligently checking every single diff the LLM makes. but let's be honest, its just so damn inviting to let it off the leash for a moment.

I think the LLM accounting benchmark is a good analogy. The first few prompts are like the first month in accounting. the books are correct before so the LLM has a good start. in the accounting benchmark then the miscalculations compound as do the terrible practices in the codebase.