Comment by refactor_master
12 hours ago
> Don't forget that very very detailed spec is actually the code
The tests, sure. But certainly not the code itself, as that sits far too close to the implementation (i.e. it is the implementation). An almost infinite number of implantations can fulfill “does foo when bar”, so how can we prove that ours is the spec itself?
It’s kind of like a scientist coming up with a hypothesis post-hoc to fit the results of the experiment.
I know you were simplifying, but "does foo then bar" is so far away from what an actual specification is that it defeats the point.
A more complete spec will capture performance requirements, input preconditions and output postconditions, error handling and recovery behaviors, threading behaviours, hardware assumptions, etc. It's hard to do these things without leaning at least somewhat on the specific language runtime you are using, otherwise you'd end up regurgitating the C standard each time you design a software system.
It's this sort of stuff that is meant when people say "sufficiently detailed".
If you're actually testing all these things, then I might agree with you that you can do it in the tests, but almost no one actually is. I'd struggle to write a test suite that tests all the specification-level assumptions I draw from my language and target platforms.
I've found LLM written unit tests to be fairly low quality which makes sense as writing good tests takes understanding that writing good code.
Also the quality of tests in general in projects is often so so and that's reflected in the output of LLMs even more so.
It'd be insane to come up with theories before doing any sort of observation.