Comment by bunderbunder
5 days ago
I've been enjoying journalist Ed Zitron's recent diatribes about how impossible it is to find a business leader who had a plan for measuring their ROI from adopting AI coding.
What he says he's consistently hearing from them mirrors what I saw at my own employer: they thought they had ROI metrics, but they actually only had usage metrics such as "lines of code committed" or "number of pull requests". The only way those could possibly work as an ROI measure is if your business charges customers by the line of code.
What they really means is they previously had no valid metric to measure productivity of developers before either. AI or not.
Measuring productivity of developers isn’t really in line with what needs to happen, either. A team can be incredibly productive and still generate negative 100% ROI if what they are building so industriously is stuff that nobody wants to buy.
Which reflects another thing I’ve seen at work. A lot of what AI coding has enabled is diving headfirst into quagmires. Our costs have spiked - not just because of the token spend, also because we gotta pay the cloud platform to run all these new services, operators to operate them, marketers to market them, etc. - but revenue hasn’t budged.
But at least pre AI, most managers presumably subjectively measured devs on relevant performance. Using systems where employees who burn the most tokens ($) per week ‘win’ is crazy - just ask the AI to spin up a subagents to implement every conceivable approach to a task, then spin up n agent judge to pick the winner, and repeat. You've immediately got 50x or whatever your previous usage from that alone.
I had cynically done this sort of tokenmaxxing for a while as a burnt offering to the token-hungry non-leadership.
Eventually I got tired of it and got back to work.