Comment by chewbacha
2 days ago
Big caveat to the productivity claim is that it’s concentrated in lesser experienced engineers and vanishes or goes negative with highly experienced engineers.
This intuitively makes sense and generally agrees with my experience. LLMs move your baseline towards the mean. If you are below average it improves and if you are above it hinders.
But also in my experience, my memory retention of the work done with an LLM is worse than doing it myself self. This leads me to believe that the lesser experienced engineers are not gaining experience!
My tinfoil hat says that this is what tech CEOs want. They want workers that are low skilled and can be paid less.
>My tinfoil hat says that this is what tech CEOs want. They want workers that are low skilled and can be paid less.
My company is small, but my opinion at the moment is the opposite. Low skill workers don’t notice the flaws of what LLMs produce, I am no longer interested in hiring any juniors, as all they do is copy and paste LLM output without thinking, which anyone could do. It is the senior engineers who are most valuable to me these days.
And as a senior engineer, I'm not interested in cleaning up vibe code. When I review a PR for a junior dev they learn from my suggestions (or I tell their boss they're not learning). Having to correct an AI is a waste of my time.
I hope you do know that your comments are straight up being pasted to an LLM that addresses or counters those, not the junior dev. They are likely learning nothing other than how to increase their PR counts for managers to see.
That's not how you should use it and LLMs aren't that bad anymore. Either you just accept the quality level of LLM code wholesale, which means accept mediocrity in exchange for volume, or you at least leverage the LLM to improve the process surrounding the production code.
The issue would be that LLMs seem to not improve senior devs productivity. In which case what you are saying is “we shouldn’t use LLMs”
Which I happen to agree with.
"The issue would be that LLMs seem to not improve senior devs productivity" - that's the opposite of what they said. And the opposite of my experience.
> My tinfoil hat says that this is what tech CEOs want. They want workers that are low skilled and can be paid less.
I think it's useful to divide companies doing software development, as well as specific jobs within a company, into two groups:
1) Companies where software is the product - where what you are working on is directly what is making the company money, and differentiating what they sell from what other companies are selling. These are the jobs you want, and where you will be valued.
2) Companies where software is only a component of the product, regarded as non-critical, or not product related at all - just internal IT systems. These are NOT the jobs you want!
In a type 1) job the company (unless it is run by idiots) recognizes that better developers = better product = more profits. They will seek out experienced skilled developers and pay them what the market demands.
In a type 2) job you are not regarded as a profit-generating asset, but rather as overhead - an expense to be minimized. The company will be looking for the cheapest, least experienced people they think can do the job. Maybe they will outsource, and/or nowadays try to use LLMs as a way to avoid needing to hire better quality developers. The development work may still in fact be demanding and require skilled developers to be done well, but if the company mindset is that developers are an expense not an asset, then they will try to get the job done with the cheapest labor regardless.
So, yeah, if at all possible don't work on things like IT systems or on products where it is not blindingly obvious to management that software quality is directly related to profitability.
this is useful and thx for that, but there are more layers and variations. In some companies, marketing CxO has the budgets and therefore the power, while minions execute the vision. In other companies, there are capital assets, physical assets and historical non-public relationships.. In those companies, the security of control, and the power of those that do the control, guide the boardroom. Software is involved in basically all of these, but common goods software from a vendor plays a large role. Just these examples from imagination lead to many permutations of "how software devs are treated in the company"
Where are you getting that from? It's a productivity tool. Someone who knows what they're doing can leverage LLMs to improve their productivity. Someone who doesn't know what they're doing will be able to increase their output but not do it as effectively and with lower quality.
If anything, it now requires more skill to maximize. If you have strong product and engineering experience you can get even more out of an LLM than one of those skills alone.
> my memory retention of the work done with an LLM is worse than doing it myself self
Do you think this is more because you are less close to the work? Or because there is simply more work being done/more to remember, so you're remembering a similar amount, but a smaller fraction?
Or both, or something else?
I believe it’s mostly due to closeness to the work and actually turning the problem over in my mind. When I use an LLM to code much of the context is lost and my understanding of the solution is diminished. The longer I use it in a code base the less and less aware of the code base I become.
This is also born out in the research which has demonstrated reduced retention of content when authoring was facilitated by an LLM. Over the long term, I believe this is widen a chasm between experienced and inexperienced engineers.
While not all CEOs will feel this way, it will not surprise me that knowledge workers will be treated as disposable, even if their work is invaluable. If they can hire a junior escort for the AI they will. That junior will also be scale goated when things go wrong.
> My tinfoil hat says that this is what tech CEOs want. They want workers that are low skilled and can be paid less. Guess it worked then. I'm happy to take a paycut to never deal with vibecode and toxic environment where every estimate is borked due to agentic coding.
People gloss over memory retention way too much. You become like a manager who knows about the overview, but not the ins and outs (to a lesser degree).
Maybe it's only me and my inexperience, but only reviewing doesn't make me as "in" as doing it myself.