Comment by Quarrelsome
18 days ago
rah, we might be in trouble here. The primary issue at play is that we don't have a reliable means of measuring developer performance, outside of subjective judgement like end of year reviews.
This means its probably quite hard to measure the gain or the drag of using these agents. On one side, its a lot cheaper than a junior, but on the other side it pulls time from seniors and doesn't necessarily follow instruction well (i.e. "errr your new tests are failing").
This combined with the "cult of the CEO" sets the stage for organisational dissonance where developer complaints can be dismissed as "not wanting to be replaced" and the benefits can be overstated. There will be ways of measuring this, to project it as huge net benefit (which the cult of the CEO will leap upon) and there will be ways of measuring this to project it as a net loss (rabble rousing developers). All because there is no industry standard measure accepted by both parts of the org that can be pointed at which yields the actual truth (whatever that may be).
If I might add absurd conjecture: We might see interesting knock-on effects like orgs demanding a lowering of review standards in order to get more AI PRs into the source.
> its a lot cheaper than a junior
I’m not even sure if this is true when considering training costs of the model. It takes a lot of junior engineer salaries to amortize the billions spent building this thing in the first place.
sure, but for an org just buying tokens its cheaper and more disposable than an employee. At least it looks better on paper for the bean counters.
Yes it's going to cause many problems forcompanies I think, but at least they will deserve it (the employees won't unfortunately unless they've drank the kool-aid, I rarely meet ICs that have drank it fwiw, which means I'm either in a serious bubble, or this is being pushed from the top down). The only clear winners are going to be chip companies.
There's never going to be an industry standard measure either. Measuring productivity as I'm sure you know is incredibly dumb for a job like this because the beneficialness of our work product can be both insanely positive and put the company on top or it can be so negative that it goes bankrupt. And ultimately a lot of what goes into people choosing whether they like the work product or not is subjective. A large part of our work is more of an art than a science and I say that as somebody that works about as far away from the frontend as one can get.