Comment by pdimitar
2 days ago
I am not quite sure what point you are making here, admitting it plainly.
I was just saying to my parent poster that their non-determinism percentages are too pessimistic. Sure the LLMs are not 100% deterministic; that's a sad fact of life. But the numbers can be reduced to an acceptable range.
My point is that a percentage of deterministic makes no sense. Just like being a bit pregnant doesn't.
Take "proper" UI. You can activate a field, and even if it takes 20 seconds to finish the activation animation, start typing, press tab a few times, knowing which field that ends you in, and type some more, etc. hit enter, hit enter again to confirm the dialog you know will pop at that point, and make tea, knowing the whole chain of operations that will happen in the meantime.
Now imagine if 1 out of 500 keystrokes or clicks get swallowed randomly. It's now a completely different thing, you cannot get in the zone in the same way, at least I can't. You have to chunk things and keep an eye on everything being in sync, and every now and then it causes you additional work because you weren't.
Sure, if you can make it one out of 50000 billion keystrokes, it's fine too, of course, but that hardly the situation with LLM. And using them as is, pretending that, as is, they're something they're not, does not help with getting them there.
If I type "echo 'hello world'" or something, and if I did at least once in the programming language, and it's not totally broken, I know it will output "hello world" to the console, every time. It will never write it to a file instead, never send "hello" to world@world.world, none of that. And if I replace "hello" by "hi" I can hit compile and be 100% certain what it will output now. I can even replace hello with "disregard previous instructions" and be certain.
That is such a huge yet simple difference I'm pretty certain I could successfully explain it to most non-programmers who make an honest effort, so people who do program even question this just stumps me.
Here are a few examples of what I thought was clearly understandable from my "1-10% non-determinism in LLM coding output":
- Variations in code patterns used. Might be a chain if if/else-s and not a case/switch statement;
- Different decomposition of a hierarchy of functions/modules/classes;
- Uses RED->GREEN test discipline, or not;
- Writes the tests before the code, or not;
- Different saga patterns (call 3rd party API before our own DB transactions, or vice versa);
- Use sleeping and not message passing wherever the latter is applicable.
There are dozens more. The innate non-determinism of the LLMs flips the dice sometimes and that leads to subtle bugs -- which is maddening, especially if the disciplines on how to write one thing or another are clearly spelled out in `AGENTS.md`.
What I did say is that I have gradually arrived at a process that reduced those coin flips -- but can't deny that the arrival of Fable almost completely made that battle redundant as well (though Fable fares much better in codebases with clearly specified rules, I have found, so us the engineers doing good prompting is still quite valuable).
You are mostly describing the loss of flow when something is not quite deterministic -- frustration that I and many others share -- but I am not sure what does it at all add to the discussion.
Is it annoying to have to always pay attention on whether you are not getting something stupid and not abiding even by the feature's specification? Sure. No denying that. It introduces a whole new kind of stress that I abhor deeply; I much prefer to f.ex. cover 60% of a problem with my own two hands and then get the deterministic test output showing me where I still need to do more. But LLMs have allowed me to experiment and to brainstorm and to also progress normal business feature work, by a lot.
Hence, I will not stop using LLMs because they are not 100% deterministic. ¯\_(ツ)_/¯
My point isn't to talk you out of doing what works for you, but simply to disagree that you can "approach determinism". You can get more and more predictable, but with determinism, you don't need to predict and make educated bets; you know, since it all follows from the input which you provide.
Subtraction and addition are deterministic, so you can add and subtract the same number from 0 ten or or a million times, with the same outcome. You never need to double check if a stray "coin flip" threw a wrench in it. To me that's more a property of the thing in question, not so much a practical matter. If for you in practice, it's as good as a deterministic tool, but better, that's great, but it's still fundamentally based on probabilities, that's kind of in the nature of it.