Comment by boredemployee

IMO anyone using any kind of metrics effectively is measuring multiple things and everything has a counterbalance. if youre measuring PR throughout you should measure release cadence, change failure rate, mean time to restore, etc.

correlating all of these let's you know if you're accelerating one activity metric but getting worse in an outcome metric - lots if slop PRs but more incidents is obviously the wrong outcome, and can be addressed by improving local verification, safer deployments, faster rollbacks, better observability, etc.

even with AI writing the code (and much more) all of these things matter. actually seeing positive outcomes (and thus RoI from AI use) requires knowing what's signal and what's noise. ive spoken to founders who see anecdotal increases in everything negative as their AI code volume goes up, and none of them are actually measuring anything effectively enough to understand what to do about it