Comment by crystal_revenge

I'm so tired of the endless stream of articles that a.) don't know what "non-determinism" means in Computer Science and b.) don't really understand how LLMs work.

"Non-determinism" (i.e. the "N" in NP-Complete) does not mean "random", it means "following all paths at once" (there's a really elegant way to understand the List monad as representing non-deterministic computing). It's such a fundamental (and delightful) concept in computer science that refusing to understand what it means is a tragedy related to the loss of technical language itself.

But, even if we just sub "non-deterministic" for "inconsistent" or "random" or "stochastic" or whatever better term you come up with. Statements like the following just seem to not understand the basic architecture of LLMs:

> Given the same input prompt, a model's output is not guaranteed to be identical on every run, at least under normal running conditions.

"Normal running conditions"? The reproducibility of the output is 100% a property of how you choose to sample from an LLM. There are virtually endless ways you can define your sampler and whether or not the same result is reproducible is a fairly minor aspect of all the other choices you make when sampling.

At the end of the day, each step of the LLM gives you a distribution over next tokens, and you can do with that what you like.

In the end the article almost becomes a tautology: the output of an LLM represents a probability distribution over all the outputs it believes are valid with some variation around an expected output. Yes, that's literally what an LLM is.