Comment by sarchertech
20 days ago
> However I guess that at least some of that can be mitigated by distilling out a system description and then running agents again to refactor the entire thing.
The problem with this is that the code is the spec. There are 1000 times more decisions made in the implementation details than are ever going to be recorded in a test suite or a spec.
The only way for that to work differently is if the spec is as complex as the code and at that level what’s the point.
With what you’re describing, every time you regenerate the whole thing you’re going to get different behavior, which is just madness.
You could argue that all the way down to machine code, but clearly at some point and in many cases, the abstraction in a language like Python and a heap of libraries is descriptive enough for you not to care what’s underneath.
The difference is that what those languages compile to is much much more stable than what is produced by running a spec through an LLM.
Python or a library might change the implementation of a sorting algorithm once in a few years. An LLM is likely to do it every time you regenerate the code.
It’s not just a matter of non-determinism either, but about how chaotic LLMs are. Compilers can produce different machine code with slightly different inputs, but it’s nothing compared to how wildly different LLM output is with very small differences in input. Adding a single word to your spec file can cause the final code to be unrecognizably different.