Comment by CharlieDigital
3 hours ago
> I don't think there's perfect code
Note I used "perfect" in my text. In this context, meaning it passes human PR reviews following our standard guidelines with minimal feedback/correction required.
> So as long as code meets or exceeds the human output, it's "good enough" and meets expectations. That's what a typical customer cares about.
Why settle for this when "perfect" is "free"? I understand this dichotomy when writing "perfect" code requires more expensive, more experienced human resources or more time so you settle for "good enough"; but this is no longer the case, is it? The cost of "perfect" is only perhaps a few fractions of a cent higher than shitty.
You only need to accurately describe what "perfect" is to the LLM instead of allowing it to regress to the mean of its training set. There really is no cost difference between writing shitty code and "perfect" code now; its just a matter of how good you are at describing "perfect" to the LLM.
For example, we very specifically want our agents to write code using C# tuple return types for private methods that return more than 1 value instead of creating a class. The tuple return type is a stack allocated value type and has a default deconstructor. We also always want to use named tuple fields every time because it removes ambiguity for humans and increases efficiency for agents when re-reading the code.
We want the code to make use of pattern matching and switch expressions (not `switch-case`) because they help enforce exhaustive checks at compile time and make the code more terse.
If we simply tell the agent these rules ahead of time, we get "perfect", consistent code each time. Being able to do so requires "taste" and understanding why writing code one way or using a specific language construct or a specific design pattern is the "right" way.
> There really is no cost difference between writing shitty code and "perfect" code now; its just a matter of how good you are at describing "perfect" to the LLM.
The consequent is at odds with the antecedent. It's a performative contradiction (if the output were truly "free", the skill of the operator would be a zero-value variable - yet, by requiring skill, you acknowledge a cost) as I prove below
> The cost of "perfect" is only perhaps a few fractions of a cent higher than shitty.
Is your cost model accounting for the cost of specification, of review and additional cycles required if review fails or the specification itself needs to be adjusted?
> If we simply tell the agent these rules ahead of time, we get "perfect", consistent code each time
No, in the simplest case, your cost of perfection is simply moving up the chain of abstraction from implementation (coding) to design and specification. In reality it also splits and moves a part of that cost downstream to verification.
This isn't some special, magical insight I have, I'm reiterating Tesler's Law right back to you.
I also encourage you to read software history - for decades it has been trivial to split out perfectly working CRUD from an ER and UML diagram, no LLM necessary. The insight is understanding why we continue to hire cheap human labor to spit out CRUD instead of using those tools.
The cost of software is, and always has been, in the figuring out the intent, not the generation of syntax.
I wish pg was more active on HN - I expect this is one of the reasons why he wanted founders to have and share the painpoints of their (potential) customers. Figuring out the intent is expensive. Mistake the intent and the best case scenario is a pivot.
Your mistake is that you think the point is that only engineers participate in the production of code. In fact, the point is that the product team and the people closest to the customer can generate the code. And for that reason, the goal is to produce a framework on top of which "perfect" code can be produced with relative ease and consistency regardless of whether the user is part of engineering or product.
This is the same cost no matter what. The LLM does not generate code on its own; some operator must provide some instruction and specification regardless so you might as well give it good ones. But here, I would point out that there is a high level of broader general instructions that incur a one-time cost of specification ("Always write this code this way").
[flagged]