Comment by delifue
3 days ago
When people refute the idea that "LLM is the new compiler from natural language to code" the common reason is that LLM is non-deterministic but compiler is deterministic.
But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And human developers are also non-deterministic. There is always the nondeterminism even before AI.
I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
For AI, if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now. To make code maintainable and debuggable you still need to read the code if the project is complex enough.
Yeah, it's not the non-determinism; it's that they don't do what I say. I can't imagine writing a nontrivial program without, at a minimum, reading and editing the pieces as they're being produced. People aren't complaining about non-determinism; they're complaining about the results being non-deterministically sometimes wrong.
And if they were deterministically sometimes wrong, that wouldn't be better.
That actually would be significantly better, because then you could work out under what conditions they were wrong and correct for it. But being stochastically wrong means you have to check everything, because you never know when the errors will creep in.
1 reply →
If the results are non-deterministic, unless I'm happy with literally any possible result, then yeah, I'm going to be unhappy with all of the other results. That's what non-determinism means. What would being right always with non-deterministic results even look like in terms of code output?
Compilers have a ton of hard optimization passes, and many of those are solved well enough via stochastic algorithms. Nobody actually fully optimizes register allocation or whatever, and a few random restarts can go a long way toward some better performance.
4 replies →
How often do programmers do what managers say, and how often do managers say what is wanted or needed?
It seems like your points are:
1. Programmers sometimes do something other than what they're told, so what's the harm in letting an LLM be disobedient instead?
2. Managers sometimes don't ask for the right things, so why not let an LLM figure things out?
To both points, those observations are literally the reason why we have software engineers writing code instead of management. The job of fully understanding the implications of all requirement interactions in excruciating detail is not a management responsibility. In the context of using LLMs as a higher-level compiler, I don't think people are arguing that management should be the people writing that higher-level code.
Separately, the failure modes are very different. If I ask a SOTA agent and a SOTA harness to construct a minimal sandbox for executing subprocesses without network access using OS sandboxing primitives to ensure you don't have Scun*horpe-esque ways of sneaking networked commands through (and assume I expanded that into a proper prompt instead of the high-level summary I'm using to convey the idea here), there's a nontrivial chance it implements some of the sandbox by string matching the commands instead of using proper OS primitives. It'll have thousands of tests showing it works, and it'll still be garbage even just from a requirements point of view and ignoring code quality or everything else we're being told to ignore when using them as higher-level compilers. There's also a nontrivial chance that if you ask either the same or a new session to check on that property it will conclude that the sandbox was implemented correctly. Good human engineers will either get the job done or tell you that they can't for some reason or another. That "looks like it succeeded but actually failed hard and you can't tell unless you read the code" error case is new to LLMs. Yeah, we write bugs from time to time (as do LLMs), but complete failures to understand the core product so badly that fixing the "bug" requires a full rewrite, one likely to introduce its own similarly catastrophic rewrite-inducing bugs, are a different beast altogether.
And again, I use LLMs a lot, almost exclusively, but they have limitations, and failing to recognize those isn't conducive to success. The frontier keeps moving, so you might find these current observations irrelevant eventually. People are choosing to not read the code today though, and that's bad.
2 replies →
> And human developers are also non-deterministic. There is always the nondeterminism even before AI.
Pointing out humans are not deterministic is irrelevant to the comparison. In the original example of LLMs replacing compilers, the deterministic compiler is being replaced, not the non-deterministic human. In the original example, we move from non-deterministic human -> sourced code -> deterministic compiler, to non-deterministic human -> sourced prompt(?) -> non-deterministic LLM. The important part is the sourced code into a deterministic final step so your build artifact doesn't change.
You can point out you can change temperature to zero or what ever, but then you don't actually get the build artifact you wanted in the first place.
> And human developers are also non-deterministic. There is always the nondeterminism even before AI.
Humans are not machines. The entire point of any technology (including specifications derived from consensus, such as the metric system, democracy, etc) is that it operates deterministically in the service of (as opposed to outside of) actual people with actual needs. This is what makes a tool a tool, and not a human. To me, it all sort of boils down to the shortest path between an intention and it's result. LLM generated code can massively shorten the time between A and B, but not necessarily shorten the path.
The feature that makes LLMs great and interesting is the same feature that makes them hallucinate. This is embedded in the entire idea of coherence being the byproduct of nudging weights in proximity as opposed to being constructed toward an unknown future. I thought it could be cool too, but a deterministic AI (zero temperature, frozen model, or whatever..) would _only_ really serve to be a leaky abstraction over its training sources. Which maybe says more about our access to information and desire to understand it.
I don't think there's a clear cut line (or even value in one!) between LLM output being good or not, but without recognizing that LLM output will always be riffing on prior inputs as opposed to being intentionally constructed toward an unknown future, we've probably lost the plot.
" if you don't ever need to read the code and AI writes perfectly bug-free program that perfectly follows instruction, then AI is a non-leaky abstraction, but that's not the case for now."
? it can never be the case.
AI has a 'noisy channel problem' and it's due to the ambiguity inherent in language.
If we have a 'way to express ourselves perfectly' in a language-like concept, at a higher level of abstraction than code, we'd have our solution.
But we don't.
We could develop really, really strong ways of writing specifications that work really well, combined with other kinds of paramterized specs, that could work - but as of now, we're not quite there.
We are 'missing a layer' - and do we make do with what we used before.
> I think the real difference is how leaky the abstraction is. If you just need to understand language instead of compiler internal to write correct code, the compiler is non-leaky abstrction. If you rely on compiler to do auto SIMD optimization, and have to do hacks to trigger the optimization, compiler is leaky abstraction.
Autovectorization is not a programming model!
I forgot who put me on that article - I think it might've been Horace He - but that is a pretty big idea in idea space for me.
I phrased your view differently in my earlier comments - we already allow "nondeterminism" (or better call it chaotic behavior) in compilers. But usually it is only limited to performance (CPU and memory consumption).
So the "LLMs are chaotic" (difficult to predict) argument is somewhat moot; the performance of GC language runtime or even modern CPU with hyperthreading is already chaotic too.
I think for many applications, people will treat correctness the same way.
> But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.). And multi-threaded compiler can be non-dterministic.
And yet, I've never used a compiler that had non-determinism I couldn't turn off, or an LLM product where Icould turn it off. I don't see the theoretical possibility for the opposite behavior as relevant if there are approximately zero examples in practice. In the real world, you can assume your compiler is deterministic and your LLM isn't, and you'll almost never be wrong. That's a real difference that abstraction is not going to fix.
[flagged]
>But the determinism isn't the real distinction. LLM inference can be made deterministic (temparaute zero, etc.).
You'll still be exposed to non-deterministic behaviour through floating point errors, GPU batching and a myriad of other factors.
Floating point math is well defined and deterministic. The results can just vary minutely between platform/hardware.
Temperature zero doesn’t mean the LLM becomes deterministic
https://mbrenndoerfer.com/writing/why-llms-are-not-determini...
If determinism is actually a goal, then these are solvable problems.
> Despite this, real-world LLMs still produce varying outputs due to factors like: > - Floating-point precision limitations
This by itself is deterministic, as per IEEE 754, as long as no parameters are "left to the implementation". The Java virtual machine had this problem before, and fixed all parameters to be deterministic. Depending on the hardware, this comes at a performance cost (if some hardware cannot support the chosen parameters) which is why the JVM introduced lenient floating-point to let the code choose between determinism or performance.
> - Hardware and parallel processing variability > - Decoding tie-breakers > - Other sampling parameters like top-k or top-p that may still introduce variation
I'm assuming this refers to different HW architectures, because hardware variability at the lowest level (silicon patterning and doping) is already dealt with by circuit designers. Again determinism can be achieved by prescribing the exact procedure, which the hardware must replicate in behavior but not necessarily in implementation. And again, this may come at a performance cost when specific hardware cannot implement the prescribed procedure at all, or only by keeping actual hardware parts unused.
--
All that said, I don't think that determinism is all that useful, because it only guarantees that the output will remain constant as long as you don't touch anything. In reality, you want the output to be "similar" as long as you only change "irrelevant" things, and determinism won't help you with that.
I don't think it is abstracting over code. It's abstracting over people. Let's do a concrete example instead of philosophizing.
"Build me a web dashboard where I see all my OSs processes etc. Maybe also memory usage and whatever, think of cool stuf. Ping me when you're done." is a perfectly valid instruction. It's vague, it's not code, but it is a perfectly valid instruction. In fact, I regularly deal with instructions from my boss that are orders of magnitude more vague and unclear than this.
What happens then is engineers scramble, gather their thoughts, present their ideas to the pointy-haired-boss and inquire about the context, the Why, the desired end state, the planning, etc. Given that "programming" is slow, extraordinarily costly and error-prone we have to carefully manage this. We create processes, systems of accountability and governance. After a "sprint" or two or ten some sort of dashboard will appear that will have about 60% of what was actually required if we are lucky (because the rest was unstated, heavily implied or genuinely unknown or some unholy combination of those things). We will have more meetings. Some will consist of stylistic discussions, some will discuss the "framework" and some "deployment". At least one will detail either the merits of a certain set of icons, the beauty of htmx or whether we should use Rust or Go on the backend. At least one person will argue for Elixir.
AI agents turn all of this into a "here is a result" and you'll be learning new requirements and iterating constructively on the idea itself instead of wasting time on NixOS and the nitty gritty of Material Design. You want Elixir? Ok sure.. let's do that in five minutes instead of twelve weeks.
I'm not at all saying the "nitty gritty" is useless or is a complete waste of time _generally_. I think I am saying that it _usually_ is a waste of time, in my bubble, and many projects would be served well with quick, concrete iterations on the ideas themselves instead of getting bogged down in bullshit. This is of course just my experience. I admittedly live in a weird bubble.
> What happens then is engineers scramble, gather their thoughts, present their ideas to the pointy-haired-boss and inquire about the context, the Why, the desired end state, the planning, etc. Given that "programming" is slow, extraordinarily costly and error-prone we have to carefully manage this. We create processes, systems of accountability and governance
That is when you go and create a whole politically motivated circus. Instead such ideas could have been coded and deployed in a single day. The actual discussion are not really about the tech, they are mostly about business needs and constraints.
The technologies are technically equivalent, but there are different tradeoffs between them. Most of engineering is finding the best set to help the business.
But it seems that a lot of people can’t bother refining their ideas conceptually. Or trying to solve the essential needs of their users. It’s like some kind of ADHD where they want “something” now, whatever the thing is.
> Instead such ideas could have been coded and deployed in a single day.
I would very much challenge this. I never saw it "in the wild". If you call this "political" then everything in life is political.
You understand that example was a simple one. You could do "something, something complex" and it wouldn't matter. You can't refine a concept you don't see. It's just that simple. We all like to think we can "know in advance" but we cannot. Time and time again we are shown to be shortsighted, ignorant and too lazy to have figured it out beforehand.
As an example us developers are confronted with this tech for years and an astonishing fraction of devs are only now starting to reckon with reality. They did not see what was right in front of them. That's just this example. It's not just us, everyone has trouble seeing ahead. We need to see the pieces moving and I assure you the clankers get pieces moving.
1 reply →
[dead]