Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).
Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).