Comment by jcoder
14 hours ago
> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history
A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
Which is why big labs have been working hard on making their harness not be stateless any longer:
https://earendil.com/posts/session-portability/
“Just take the session thread to another provider” might not be feasible anymore soon-ish.
While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow.
The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?
If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
Soooo… when can we expect an encrypted handoff.md to fully prevent session portability, then?
(Only half-/s)
And this is why you should not use the model provider's harness. It is to tight of a coupling of services that's only business purpose is lock-in.
Knower
Ding ding ding
Which is why folks should be jumping to pi / oh my pi as soon as possible
This is also why anthropic no finger lets you use the flat rate sub in non-claude-code harnesses
(They do but you are charged at api rates)
As a heavy pi user - this doesn’t help, for codex. The thinking traces are encrypted and you cannot move that session over to eg your local ds4. It will lack those precious thinking traces that you absolutely need to continue a session productively.
They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.
This is harder to do on other architectures that themselves aren't fully deterministic though.
Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://news.ycombinator.com/item?id=45200925
Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.
You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
If you send the seed along, you can have a non-zero temperature: when it's time to select tokens, a PRNG will be seeded and used for the selection.
Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
Perhaps they just used the exact same random seed everywhere?
I appreciate the replies on the determinism point and I’ve learned some new things here. In any case I probably should not have tagged that on, as my main point was to share that the sort of property that parent is talking about (whether true for all LLMs/providers/harnesses or not) is statelessness, not idempotency.
LLMs are, in theory, deterministic. Sampling is not intrinsic to LLMs.
Greedy decoding a single batch in most libraries will give you mostly deterministic outputs. Higher batch sizes can increase variance.
But all of this is down to CUDA and/or kernel implementation issues.
It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.
According to [1] there is - unfortunately - increasingly more state in LLM sessions.
1: https://earendil.com/posts/session-portability
>LLM responses are certainly not idempotent, as they are not even deterministic.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.
It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.