Comment by rbehrends
1 day ago
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
it also turns out that a non-ambiguous spec, is actually indistinguishable from code
Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).
I'm sorry but if you can finish spec'ing via 10 minutes of interview style questions, you're building to-do apps or a bloons TD clone not the next thing that replaces search engines.
Not sure it bolds well for these AI companies if GP is seeing the LLMs fall short when all they are making are “to-do apps”?