Comment by electroly
7 hours ago
LLMs are great at reviewing. This is not stupid at all if it's what you want; you can still derive benefit from LLMs this way. I like to have them review at the design level where I write a spec document, and the LLM reviews and advises. I don't like having the LLM actually write the document, even though they are capable of it. I do like them writing the code, but I totally get it; it's no different than me and the spec documents.
Right, I'd say this is the best value I've gotten out of it so far: I'm planning to build this thing in this way, does that seem like a good idea to you? Sometimes I get good feedback that something else would be better.
If LLMs are great at reviewing, why do they produce the quality of code they produce?
Reviewing is the easier task: it only has to point me in the right direction. It's also easy to ignore incorrect review suggestions.
Imho it's because you worked before asking the LLM for input, thus you already have information and an opinion about what the code should look like. You can recognize good suggestions and quickly discard bad ones.
It's like reading, for better learning and understanding, it is advised that you think and question the text before reading it, and then again after just skimming it.
Whereas if you ask first for the answer, you are less prepared for the topic, is harder to form a different opinion.
It's my perception.
Its also because they are only as good as they are with their given skills. If you tell them "code <advandced project> and make no x and y mistakes" they will still make those mistakes. But if you say "perform a code review and look specifically for x and y", then it may have some notion of what to do. That's my experience with using it for both writing and reviewing the same code in different passes.