Comment by SpicyLemonZest
5 days ago
I'm not sure the leaders would disagree with what you're saying. They tokenmaxxed to understand what it looks like when AI gets into every corner of the business; now they feel they've gotten enough info (or at least that more info wouldn't be worth the cost), so they're adding in cost controls. As the article says, this is not great for AI model providers trying to predict what their future revenue is going to be, but it's not obvious that there's any mistake here for AI users.
> They tokenmaxxed to understand what it looks like when AI gets into every corner of the business
Perhaps that is what they were trying to do, but the reality is that all they will have got is a large token bill. The decision makers may have hoped that tokens would be used in most productive fashion possible so they could evaluate if the cost was worth it, but what they will have actually got is what they asked for and measured, high token usage (applied to whatever people needed to do to get their usage stats up, regardless of productivity).
The other business-as-usual factor is that there will be false reporting up the chain, so if the company understands the CEO want to see high AI usage and productivity gains, then s/he will see high AI usage (a large token bill) and will be fed success reports of corresponding productivity gains.
In a typical corporate environment, if all your peers are reporting success, achieving what the CEO wants, do you want to be the only one reporting failure? So - everyone reports high AI usage (easy for the employees to make happen), and most everyone also reports productivity gains if they understand this is the expectation.
I’m imagining a lot of programmers suddenly being given the impossible task of reporting what worked and what didn’t, and middle management making up some retrospective evaluation with fat PowerPoint decks and meaningless graphs in an effort to present to C-levels some measures of success other than token use.
As the saying goes "figures can't lie, but liars can figure".
If you want to report productivity gains or cost savings from some initiative (increased AI usage or whatever) and need some stats to point to, then you just point to whatever is working, for whatever reason, and attribute the success to the new initiative.
In a company I used to work for, one manager, when pushed to increase machine learning usage (a few years back, before ML became AI), just renamed his product from foo to foo-ML (with ZERO ML usage), and reported how well it is working. He has since been promoted twice.
It’s not clear companies were measuring anything but token usage. What information could leadership have collected to determine what worked, what didn’t, and what needs more data? Other than the balance sheet and revenue, do companies actually have sufficient information to understand the results?
Were they trying to measure other things? Definitely. The COO at Uber, one of the examples in the source article, has talked publicly about how they've searched for (and so far failed to find) a link between micro-level metrics driven by AI and concrete improvements in high level project velocity.
Do these measurements have sufficient information? As much as any, I'd guess. It sounds like you already know that it's pretty hard in general to measure the productive output of software development organizations.
I have no doubt a few companies, like Uber, were measuring other things and had applicable metrics in place before adopting Clod or CoPilot or whatever automation. I'm speaking in the general sense of companies adopting the latest hype without reflection.