Comment by darkwi11ow
17 hours ago
> Whereas if you're handing off all dev and no longer reading code, just making CTO-level architecture decisions, then there are only 2~3 models right now worth using, none cheap.
I use frontier models every day and night, but not sure if the described approach is possible with any of them right now. Models still make stupid mistakes and people who do not read the code produced are bound for surprising discoveries in their codebase. Not trying to downplay the effectiveness of LLM-assisted coding and all that, but it's hard to imagine how much stupid shit i would've committed if I was not reading their output.
Serious question: Are you do a combative code analysis with a range of different models?
I used to find real dumb bugs in code reviews. With 2-3 different models doing a combative code review.. most of that is already fixed when I get it.
Yeah, I have extensive collection of prompts for reviews and usually run 3-5 (automated) review takes on large sets of changes before doing eyeball reviews. Auto reviews indeed find many obvious bugs, but they usually slip through things that are less obviously stupid but are bad conceptually, for example, using persistence / queue / repository in a place where set of pure functions would be enough; or vice-versa, not persisting the state that must be persisted for reasons, etc.
Another problem is that review changes can easily introduce new issues, so my review prompts go like that: if you find outright shit, fix it right away, but if you have slightest concern, just write your ideas and suggestions in the review and let me decide then. I try from time to time to omit the "slightest concern" part and make agent do all the shit it wants, and usually end up reverting those changes.
But as the sibling comment said, not all code needs quality and indeed I skip some code too, like user playwright journeys -- I only need one auto review pass to verify they indeed test things according to a doc, and I'm fine not reading them. Not all code like that though, some stuff would wreck havoc later if its not looked at.
If you can test well enough, both manually and automated, then it's okay if there's a bunch of stupid mistakes in the code, as long as the end result works right. Not unlike biology.
If you view creating programs like an optimization problem over the feasible region of valid programs, this is not unreasonable.