Comment by rbongers

AI is great at reviewing code. It is not great at reviewing intention. If the prompt was bad, not understanding or thinking about how to implement the feature properly, the review usually doesn't point out exactly what's wrong. This can lead to a sort of "polishing a turd" approach to code reviews where you patch some of the negative side effects for an implementation that is completely wrong.

This is a result of AI still being bad at telling the prompter "no". Not only because it will blindly reproduce what you tell it to while writing code without pushing back on assumptions without asking, but because it will often blindly accept an implementation approach while reviewing.

I suspect this is a hard problem, because if AI stops listening at the wrong time it can be a massive inconvenience.

I have noticed that AI is slowly getting better at pushing back on different types of things each time the frontier models get better though. I've also noticed on reviews that it will tend to hallucinate or nitpick around the problem. I assume it sees something wrong there, starts generating output, and points out something different there instead because there is pressure to be both agreeable and helpful, which are not always compatible.

I think it is getting better, but it can't always see the forest for the trees.