Comment by jerf
4 hours ago
One of the things I've started noticing at work is that for the last few months I was banging away at having multiple streams at a time. But over time, I've reached the point where there is some human blocker on the stream, be it QA to test it, other humans to have attention to pick it up for the next step, deployment of some sort, or in some happy cases, it made it through to deployment and the problem is now solved. In the past couple of weeks my multistreaming is toning down and more often I'm back to one stream of work. Often a much higher level stream than I would have been working at a couple of years ago, and I'm moving faster in a number of ways. Some of that "faster" has been burned up in things like changing environments that I previously just wouldn't have done, though.
Open source has had a QA problem pretty much its entire life... not in the sense that it stinks necessarily, but in that getting it at all is a challenge and was in many projects probably already the blocker to further progress.
One wonders if what is happening here is just that even in the software space, where the fluidity and flexibility is maximized as we work with nearly entirely digital constructs, that once we burn through the dry tinder of "all the stuff we know we needed but didn't have the coding time to get to" and the frontier of "work to do" advance on all fronts to "some sort of human or real-world blocker", it may just turn out that the steady state improvement in productivity after accounting for Amdahl's Law, even when code is "nearly free", is only a maybe 2x or 3x improvement after all. If this is true, goodness help the financial bubble which is assuming that something like 5x or 10x indefinitely is a lower bound, if even in the best possible domain for AI that's all we really get.
We might be producing shovelfuls of code at 10-20X the previous rate, but we're not producing value proportionally, maybe we are even inversely proportionally.
At most places I've worked, the "designing and writing code" part (basically the part that LLMs can do) was maybe 5-15% of the calendar time it took to actually get the software out into the market. The remaining 85-95% of the time was spent on squishy people-things like aligning stakeholders, getting executive buy-in, working with x-functional teams, getting priorities aligned so one team wasn't done in 30 minutes while the other team took 30 days, doing security reviews, doing privacy reviews, doing legal reviews, doing QA and verification, lining up marketing, educating salespeople, waiting for people to get back from vacation, waiting for people to get back from sick leave, waiting for people to respond to their E-mails, waiting for people to click "approved" on various internal review sites, doing final reviews with executives so they know what their company is doing. These other things take up the vast majority of the time it takes to release a feature. LLMs are only speeding up the relatively tiny 'code writing' part.
I think 2x or 3x is maybe too generous. Obviously pure code output is extremely high but I would think for something to truly be a 2x a company would either have to be able to effectively get rid of half its SWE workforce or produce enough to make some notable increase in revenue.
It doesn't seem like the SWE labor market has been shedding anywhere near that order of magnitude of jobs (there are about 1.7 million SWEs in the USA alone) or that companies going heavy into AI are boosting revenue significantly.
I think all things considered we're only getting a single digit percentage total productivity increase.