Comment by n4r9
1 day ago
There's been a lot of talk about the purpose of code review recently. It makes sense in the face of AI. Heres a link that was submitted a little while ago: https://mathstodon.xyz/@mjd/115096720350507897
And in response I wrote a non-exhaustive checklist of things that a code review can look for:
- Does it functionally achieve what it sets out to (as per tacker issue or PR description)?
- Does it have extraneous code? Leftover debug prints, private API keys etc...
- Does it have any obvious defects? Memory leaks, un-handled edge cases, security flaws, obsolete API calls, etc...
- Could it be more understandable? Add/remove abstractions, better variable/method names, more/less functional etc...
- Is the style consistent with the codebase and/or style guidelines?
- Are there obvious performance improvements? Hashset instead of list, lazy evaluations, etc...
- Is it sufficiently well tested?
I think LLMs are okay at most of these, and worst at the first.
Missing my biggest issues as you ask the agents to do larger tasks with less up front planning.
Is there already a pattern or code on in in the existing codebase that handles this functionality,
Do we really need net new code to achieve this functionality?
Can existing code be extended or abstracted to more cleanly implement this feature or functionality.
“Net new” is one it seems to be particularly bad at.
I don’t think I have ever even once seen an LLM solve a problem related to overengineering by simply removing the overengineering. They always choose to add more epicycles and further compound the complexity.
You can tell it to do that. I don't mean abstractly, I mean concretely, tell it how to solve it and it will do what you tell it.
2 replies →
- Do we want this? Cost/Benefit etc
- Is the change architecturally right?
Particularly the latter LLMs seem still pretty useless at.
The former feels more like a product leadership problem.
Although I do think that LLMs have made it much easier to justify writing low-value code which can make this more common now.
The thing is, leadership relies on the people actually building the software to provide concrete, accurate feedback about cost. Without that they have no chance to do a decent cost/benefit analysis.
But AI has engendered a collapse in developers’ ability to actually do that. Those of us who are stuck on the vibecoding bandwagon have lost the comprehensive understanding of the systems under our care that we need to understand and explain the quality and maintenance implications of a change.
Worse, if you happen to lose your mind and suggest the initial development cost is anything more than ~zero, your friendly neighborhood Claude keener will publicly shame you for not having sufficient faith in the Glorious Agentic Future. Product leadership will then have no choice but to side with them, not necessarily because they agree, but because they, too, are aware that we’re still in the phase of the hype cycle where openly questioning said hype is a career-limiting move.
The "do we want this" question can also apply to functionality, not just cost/benefit. I've seen AI volunteer "features" that aren't actually useful or are actively harmful to user experience but it generates them because they do make sense in different contexts.
I work on an open source project, so to-be-reviewed work can come in without any involvement by anyone :)
Code review also transfers knowledge to the reviewers!
I think we're kind of missing a layer of testing, that should sit above unit and integration tests.
Something akin to "meta-tests", which are not about testing the code itself, but the approaches taken by the implementation - i.e. architecture, understandability, terseness, etc.
These tests would operate on the source code level, even when testing code for a compiled language.
> I think LLMs are okay at most of these, and worst at the first.
LLMs are worst at not realizing problems that I'd call "meta" problems. Here's one example to illustrate it:
I was allowed by my employer to work on a small project within the large collection of the projects which all constitute the product the company sells. Like a few dozens of other projects, it's written in Python. The company doesn't have any explicit policies about how Python projects have to be organized, it requires testing, linting, a CI code to package it etc, but the guidelines are very permissive. It just so happens that, beside the guidelines, there's a tradition: every other Python project in my company uses the typical Python bloatware, like masonry with a lot of insanity and mental flips going on in pyproject.toml, which is, in general, very typical for Python community at large.
My project used none of that. Instead, I wrote a ~100 lines setup.py file (no dependency on setuptools/distutils) that assembles the wheel and runs project maintenance tasks in the same way (interface-wise) things used to work decade or two ago (eg. "./setup.py test" if you want to run unit tests).
The AI reviewer didn't bat an eyelash. Found some typos in the comments, a problem with Base64 formatting, and generally OK'd the whole thing.
I knew I was on my way out. And I generally enjoy seeing people having a fit of rage when they know they are wrong (especially, together with many more like them), and scrambling for arguments that they know to be lies. I felt a little bit vindicated for the years of suffering I had to endure working with what might have been the dumbest and the most entitled manager I had in my life. :D
Anyways. My point is: the AI caught none of it. It was very happy with my approach to Python project management.
* * *
While my story is... more of an odd case, where this does have much wider implications is the AI-generated code. AI-generated code often fails to match these meta-requirements. I've seen AI reviewer OK'ing a PR containing AI-generated 10K loc Python file. I human would probably break after reading the first 1K lines. But AI doesn't get "tired", it just kept picking on typos in comments, criticizing short variables names etc. And there are other aspects in which AI-generated code is weird to humans in the ways that humans simply won't accept it, but AI reviewer would completely ignore as non-issue.