Comment by alsodumb
1 day ago
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
it also turns out that a non-ambiguous spec, is actually indistinguishable from code
Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).
I'm sorry but if you can finish spec'ing via 10 minutes of interview style questions, you're building to-do apps or a bloons TD clone not the next thing that replaces search engines.
Not sure it bolds well for these AI companies if GP is seeing the LLMs fall short when all they are making are “to-do apps”?