Comment by vidarh
1 month ago
Because very few knows how to use AI. I teach AI courses on the side. I've done auditing supervised fine tuning and RLHF projects for a major provider. From seeing real prompts, many specifically from people who work with agents every day, people do not yet have the faintest clue how to productively prompt AI. A lot of people prompt them in ways that are barely coherent.
Even if models stopped improving today, it'd take years before we see the full effects of people slowly gaining the skills needed to leverage them.
Sure there are people holding it wrong.
But there are thousands of people on social media claiming huge productivity gains. Surely at least 5% of devs are holding it right.
If a 10x boost is possible, we’d notice that. There are only 20k games a year released on steam.
If my hypothesis is true and the real final output boost is somewhere near 20%, we’re seeing exactly what you’d expect.
I'd love to look at what you consider to be good prompts if you could provide a link.
You'd be surprised how low the bar is. What I'm seeing is down to the level of people not writing complete sentences.
There doesn't need to be any "magic" there. Just clearly state your requirements. And start by asking the model to plan out the changes and write a markdown file with a plan first (I prefer this over e.g. Claude Code's plan mode, because I like to keep that artefact), including planning out tests.
If a colleague of yours not intimately familiar with the project could get the plan without needing to ask followup questions (but able to spend time digging through the code), you've done pretty well.
You can go over-board with agents to assist in reviewing the code, running tests etc. as well, but that's the second 90%. The first 90% is just to write a coherent request for a plan, read the plan, ask for revisions until it makes sense, and tell it to implement it.
Not surprising. Many folks struggle with writing (hence why ChatGPT is so popular for writing stuff), so people struggling to coherently express what they want and how makes sense.
But the big models have come a long way in this regard. Claude + Opus especially. You can build something with a super small prompt and keep hammering it with fix prompts until you get what you want. It's not efficient, but it's doable, and it's much better than having to write a full spec not half a year ago.
2 replies →
> Just clearly state your requirements.
Nothing new here. Getting users to clearly state their requirements has always been like pulling teeth. Incomplete sentences and all.
If the people you are teaching are developers, they should know better. But I'm not all that surprised if many of them don't. People will be people.
2 replies →
One thing that I’ve often seen is models, when very much told to just write a plan, still including sizeable amounts of code in the plan.
Maybe it’s needing to step back and even ask for design doc before a plan, but even then…