Comment by trashb
18 days ago
Those are some high profile (celebrity) developers.
I wonder if they have measured their results? I believe that the perceived speed up of AI coding is often different from reality. The following paper backs this idea https://arxiv.org/abs/2507.09089 . Can you provide data that objects this view, based on these (celebrity) developers or otherwise?
Almost off-topic, but got me curious: How can I measure this myself? Say I want to put concrete numbers to this, and actually measure, how should I approach it?
My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.
So how would I actually go about and be able to provide data for this?
> My naive approach would be to just implement it twice, once together with an LLM and once without, but that has obvious flaws, most obvious that the order which you do it with impacts the results too much.
You'd get a set of 10-15 projects, and a set of 10-15 developers. Then each developer would implement the solution with LLM assistance and without such assistance. You'd ensure that half the developers did LLM first, and the others traditional first.
You'd only be able to detect large statistical effects, but that would be a good start.
If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.
The initial question was:
> wonder if they have measured their results?
Which seems to indicate that there would be a suitable way for a single individual to be able to measure this by themselves, which is why I asked.
What you're talking about is a study and beyond the scope of a single person, and also doesn't give me the information I'd need about myself.
> If it's just you then generate a list of potential projects and then flip a coin as to whether or not to use the LLM and record how long it takes along with a bunch of other metrics that make sense to you.
That sounds like I can just go by "yeah, feels like I'm faster", which I thought exactly was parent wanted to avoid...
7 replies →
> I wonder if they have measured their results?
This is a notoriously difficult thing to measure in a study. More relevantly though, IMO, it's not a small effect that might be difficult to notice - it's a huge, huge speedup.
How many developers have measured whether they are faster when programming in Python vs assembly? I doubt many have. And I doubt many have chosen Python over assembly because of any study that backs it up. But it's also not exactly a subtle difference - I'm fairly 99% of people will say that, in practice, it's obvious that Python is faster for programming than assembly.
I talked literally yesterday to a colleague who's a great senior dev, and he made a demo in an hour and a half that he says would've taken him two weeks to do without AI. This isn't a subtle, hard to measure difference. Of course this is in an area where AI coding shines (a new codebase for demo purposes) - but can we at least agree that in some things AI is clearly an order of magnitude speedup?
It is entirely plausible that your colleague did have a significant speedup from using AI. But that may also be a one off overstated story (probably not as you respect his judgement as "senior dev").
I'm not saying they didn't get a speedup or you can't get a speedup. I'm just wondering how they determined that speedup and how reliable they are as a source for the statement "using AI will improve your performance".
As the other thread was discussing, it is a difficult thing to prove but some (debatable) measurement proves more then anecdotal stories. And I am trying to probe for some sources where these kind of statements are proven as I am a programmer continually judging the tools available to me.
And on the flip side there is at least some evidence that programmers often overstate/overestimate their speedup when using AI.