Comment by davidpapermill
10 hours ago
I've heard something along the lines of "Claude is like a compiler: source code is the new object code, you don't look at that anymore" many times.
And I don't really think this is true. Compilers are usually deterministic, and whilst we can find edge cases, it's nothing like an AI agent writing all the code for you.
I think you have two choices, given the Claude is a code generator and not a compiler: (a) you review most or all of code to make sure it makes sense, or (b) you trust but verify via a strong test suite, potentially also created by Claude.
The problem with (a) is that you lose a lot of the speed-up. The problem with (b) is that you have no human oversight and the code may be incomplete, badly designed, or plain wrong.
Currently we review all code because correctness is extremely important to what we do, but that comes at a cost.
I don't know what the answer is here, in general. Does trust build over time? Do the models just get so good we can trust them to make zero mistakes?
Part of the value of LLMs and humans is nondeterminism. Pair nondeterministic output with strictly verified results (proper tests) and you can create a useful working system.
Humans can't build a system perfectly, and agents definitely can't. And agents (like humans) will build a different system every time even with the same prompt. Even if it's just trivial differences like array vs linked-list, there are still differences.
The value in having executable code is that it won't change its behavior unless you deliberately modify it. I don't think there are any shortcuts. Models will get smarter and smarter and make fewer mistakes, but you'll still want to produce real source code to execute because it's consistent (you can sell it as a product, set it and forget it, etc.), and most importantly:
running real code is orders of magnitude faster than having an AI either run the process "manually", or have it write the code out multiple times.