Comment by embedding-shape
16 days ago
> there isn't really any "prompt engineering" involved
You should make an experiment; take someone who never used any LLMs or agents, and tell them to use it for the first time in front of you, and tell them to build something like a calculator program or whatnot. Bonus points if they're ICs or at least not-managers.
I think there is a lot us engineers take for granted, when it comes to communicating via text, how to state things clearly and what we think/reason when we read things. A lot of people don't have those "skills" innate, and the first time they use LLMs, they basically don't know how to interact with them, until they realize what they're able to do and not. Then they also learn what to say to steer the model into the right way, this is quite literally a "prompt engineering" skill they're now learning.
You don't even have to go outside engineers. I have teammates that get very little out of Claude Code because the way they integrate their own knowledge doesn't allow them to think of what Claude might not know. They'd say a task was impossible with the tooling, and I'd get instant answers, because I understand what is weird internal business logic sitting 6 repos away, and what is knowledge claude has by default. I can commit Claude.md files for them, but I have to include EVERYTHING, because otherwise they'll let Claude make assumptions and waste minutes, if not hours.
It's a big part of what, in my experience, is separating the very good engineer from the iffy one: Do you have a good mental model, and can you put yourself in the shoes of people sitting in a different mental model? It makes you a better dev, and even more so when it comes to AI tools, which have their own kind of alien brain.
Coding LLMs are distilling developers. It's like the old experiment where you have someone write down the steps to make pancakes and they don't tell you to crack the eggs before adding them to the batter: it takes a particular mindset to be able to make a model of what is supposed to happen and deconstruct that to the level appropriate for implementation.
Until now, the actual act of writing code: terminology, syntax, etc. was a significant hurdle, and that underlying mindset was a very useful, but missing in a surprisingly large number of developers, skill.
Now with LLMs doing the work of "translate this into code," increasingly the only thing that matters is that exact ability. And developers that don't have it or can't develop it won't be developers for long.
or when LLMs won't be able to run on non-existing money any more the scenario will be the opossite.
Thanks for putting into words what I have been seeing a lot at work and haven't been able to put my finger on. We tend to have quite diverse _workflows_ between devs at my company, and success seems to correlate with injecting better context earlier in the process.
I like to chat with Claude about how to approach a given problem, bring in extra context, etc, before even really drafting up a plan, while other people dive into implementation immediately and go on wild goose chases.
90% of the time we end up in the same place in roughly the same amount of time, and there are obviously tradeoffs to spending more time planning vs implementing. I'm oversimplifying as well.
I couldn't agree more. Socratic methodologu, domain modelling, systems thinking, pipes-and-arrows problem solving etc. These are the skills that get real work done in coding agents these days.
This makes a lot of sense and explains why some people are so captivated by modern models, while others see progress as merely incremental.
I'm sure that explains some of it but I really don't think it explains most of the people who have been AI-pilled in the last nine months. There was no amount of context I could give GPT-4o that would make it a net benefit to use that for agentic development. I tried it with quite sophisticated prompt systems and much simpler ones, compendiums of code & business analysis and sparser ones. Yet it just wasted my time - still there were people using Cursor with that model and saying it was life changing. I didn't have that experience until Opus 4.5 - its possible I could have had it earlier but that was when I happened to try it again.
I think many of the people who have become "AI Pilled" (I'll include myself here) had it happen in the last 3 months. Even over the Christmas break, when the Wiggums loop got so much coverage - I still wasn't that blown away going into January/February- 50%+ of the time I'd just write the code myself. I like coding.
But - I don't know if it was April, or May - but very recently - the coding harnesses paired with decent SOTA models like Opus 4.8/GPT 5.5 - just started showing a lot more consistency, and completeness, and sometimes downright clever behavior - that they started to become way more useful.
Just one out of hundred+ examples - I gave Claude Code (Opus 4.8 High) a complex task that involved consul, vault - but I had neglected to give it sandbox permission to download from hashicorp.com. So - it created a entire test harness that simulated both the behavior of Vault and Consul - created all it's test cases, verified that they passed - and when I came back 40 minutes later said that it was all done.
It's test harnesses so accurately simulated the behavior of Vault/Consul - that on first try - no refactoring whatsoever - all of the protobuf/AESGCM/API behavior (that has varied significantly between versions) - worked.
This was something that would have taken me, someone super super familiar with the code and tools and APIs - a minimum of 3 solid days of work - and that would likely involve hundreds of attempts and refactors as I unwound all the weird encryption and packaging layers. It zero-shotted a full solution without having an API to test against
If these agents actually have an actual test-harness - It's honestly hard to imagine what they can't do - subject only to imagination and budget at this point.
Speaking personally - something changed Between January and, Let's say May - in which instead of seeing these things as mostly interesting technology demonstration, in which the flaws outweighed the benefits - I now genuinely think they are the future of programming. I'm dubious that I'll write much software manually in the future - beyond what I do for personal pleasure.
3 replies →
Which way do you think that goes? Are the ones who "get it" the ones who are captivated or see them as incremental?
I guess all of them?
Some people "got" LLMs back in 2022, others needed it to evolve a bit.
It's not unlike computers. I started using them back in the 90s and absolutely nobody I knew was interested, while today everyone carries one in their pockets...
By that same logic (and I’m agreeing with you as of now), engineers shouldn’t get too comfortable treating “being good at text communication” as a lasting edge. With how quickly agentic coding is evolving, it’s worth considering the possibility that many of the prompting and steering skills we view as valuable today could become far less important in a matter of weeks or months.
Recently I have the SEO guy governing the mostly static, public site with Claude Code. He loves it but you would never imagine the level of mental illness Claude comes up with. If it were an employee I’d literally throw him out the front door, labor laws be damned. And as always, every insane thing it does is some direct echo of its concept and training.