Comment by llmslave2
3 days ago
> Even with refinement and back-and-forth prompting, I’m easily 10x more productive
Developers notoriously overestimate the productivity gains of AI, especially because it's akin to gambling every time you make a prompt, hoping for the AI's output to work.
I'd be shocked if the developer wasn't actually less productive.
For personal projects, 10x is a lower bound. This year alone I got several projects done that had been on my mind for years.
The baseline isn't what it would have taken had I set aside time to do it.[1] The baseline is reality. I'm easily getting 10x more projects done than in the past.
For work, I totally agree with you.
[1] Although it's often true even in this case. My first such project was done in 15 minutes. Conceptually it was an easy project. Had I known all the libraries, etc out would have taken about an hour. But I didn't, and the research alone would have taken hours.
And most of the knowledge acquired from that research would likely be useless.
I accept there are productivity gains, but it's hard to take "10x" seriously. It's such a tired trope. Is no one humble enough to be a meager 2.5x engineer?
Even 2.5x is absurd. If they said 1.5x I might believe them.
I'm building an AI agent for Godot, and in paid user testing we found the median speed up time to complete a variety of tasks[0] was 2x. This number was closer to 10x for less experienced engineers
[0] tasks included making games from scratch and resolving bugs we put into template projects. There's no perfect tasks to test on, but this seemed sufficient
8 replies →
I estimated that i was 1.2x when we only had tab completion models. 1.5x would be too modest. I've done plenty of ~6-8 hour tasks in ~1-2 hours using llms.
1 reply →
I recently used AI to help build the majority of a small project (database-driven website with search and admin capabilities) and I'd confidently say I was able to build it 3 to 5 times faster with AI. For context, I'm an experienced developer and know how to tweak the AI code when it's wonky and the AI can't be coerced into fixing its mistakes.
3 replies →
10x probably means “substantial gain”. There is no universal unit of gain.
However if the difference is between doing a project vs not doing is, then the gain is much more than 10x.
I don't know what to tell you, it's just true. I have done what was previously days of BI/SQL dredging and visualizing in 20 minutes. You can be shocked and skeptical but it doesn't make it not true.
There is no x is because LLM performance is non deterministic. You get slop out at varying degrees of quality and so your job shifts from writing to debugging.
From one personal project,
Last month:
Last 7 days:
This has a lot of non-trivial stuff in it. In fact, I'm just about done with all of the difficult features that had built up over the past couple years.
Don't worry, it's an LLM that wrote it based on the patterns in the text, e.g. "Starting a new project once felt insurmountable. Now, it feels realistic again."
That is a normal, run of the mill sentence.
Yes, for an LLM. The good thing about LLMs is that they can infer patterns. The bad thing about LLMs is that they infer patterns. The patterns change a bit over time, but the overuse of certain language patterns remains a constant.
One could argue that some humans write that way, but ultimately it does not matter if the text was generated by an LLM, reworded by a human in a semi-closed loop or organically produced by human. The patterns indicate that the text is just a regurgitation of buzzwords and it's even worse if an LLM-like text was produced organically.
I can't prove it of course but I stand by it.
1 reply →
I think it depends what you are doing. I’ve had Claude right the front end of a rust/react app and it was 10x if not x (because I just wouldn’t have attempted it). I’ve also had it write the documentation for a low level crate - work that needs to be done for the crate to be used effectively - but which I would have half-arsed because who like writing documentation?
Recently I’ve been using it to write some async rust and it just shits the bed. It regularly codes the select! drop issue or otherwise completely fails to handle waiting on multiple things. My prompts have gotten quite sweary lately. It is probably 1x or worse. However, I am going to try formulating a pattern with examples to stuff in its context and we’ll see. I view the situation as a problem to be overcome, not an insurmountable failure. There may be places where an AI just can’t get it right: I wouldn’t trust it to write the clever bit tricks I’m doing elsewhere. But even there, it writes (most of) the tests and the docs.
On the whole, I’m having far more fun with AI, and I am at least 2x as productive, on average.
Consider that you might be stuck in a local (very bad) maximum. They certainly exist, as I’ve discovered. Try some side projects, something that has lots of existing examples in the training set. If you wanted to start a Formula 1 team, you’re going to need to know how to design a car, but there’s also a shit ton of logistics - like getting the car to the track - that an AI could just handle for you. Find boring but vital work the AI can do because, in my experience, that’s 90% of the work.
Mmm, I do a lot of frontend work but I find writing the frontend code myself is faster. That seems to be mostly what everyone says it's good for. I find it useful for other stuff like writing mini scripts, figuring out arguments for command line tools, reviewing code, generating dumb boilerplate code, etc. Just not for actually writing code.
I’m better at it in the spaces where I deliver value. For me that’s the backend, and I’m building complex backends with simple frontends. Sounds like your expertise is the front end, so you’re gonna be doing stuff that’s beyond me, and beyond what the AI was trained on. I found ways to make the AI solve backend pain points (documentation, tests, boiler plate like integrations). There’s probably spaces where the AI can make your work more productive, or, like my move into the front end, do work that you didn’t do before.
Numbers don't matter if it makes you "feel" more productive.
I've started and finished way more small projects i was too lazy to start without AI. So infinitely more productive?
Though I've definitely wasted some time not liking what AI generated and started a new chat.
> Numbers don't matter
Yes that's already been well established.
It does matter because you're still using up your life on this stuff.
Just as a personal data point, are you a developer? Do you use AI?
Yes and yes.
And you find yourself less productive?
8 replies →
One of my favorite engineers calls AI a "wish fulfillment slot machine."
Username checks out
> I'd be shocked if the developer wasn't actually less productive
I agree 10x is a very large number and it's almost certainly smaller—maybe 1.5x would be reasonable. But really? You would be shocked if it was above 1.0x? This kind of comment always strikes me as so infantilizing and rude, to suggest that all these developers are actually slower with AI, but apparently completely oblivious to it and only you know better.
I would never suggest that only I know better. Plenty of other people are observing the same thing, and there is also research backing it up.
Maybe shocked is the wrong term. Surprised, perhaps.
There are simply so many counterexamples out there of people who have developed projects in a small fraction of the time it would take manually. Whether or not AI is having a positive effect on productivity on average in the industry is a valid question, but it's a statistical one. It's ridiculous to argue that AI has a negative effect on productivity in every single individual case.
1 reply →
We’re seeing no external indicators of large productivity gains. Even assuming that productivity gains in large corporations are swallowed up by inefficiencies, you’d expect externally verifiable metrics to show a 2x or more increase in productivity among indie developers and small companies.
So far it’s just crickets.