People are describing the results when they say models are non-deterministic. Give it the same exact input twice, and you'll get two different outputs. Deterministic would mean the same input always gives the same output.
Even then, depending on the specific implementation, associativity of floating point could be an issue between batch sizes, between exactly how KV cache is implemented, etc.
People are describing the results when they say models are non-deterministic. Give it the same exact input twice, and you'll get two different outputs. Deterministic would mean the same input always gives the same output.
Yes. Look up LLM "temperature" - it's an internal parameter that tweaks how deterministic they behave.
The models are deterministic, the inference is not.
Which is a useless distinction. When we say models in this context we mean the whole LLM + infrastructure to serve it (including caches, etc).
What does that even mean?
Even then, depending on the specific implementation, associativity of floating point could be an issue between batch sizes, between exactly how KV cache is implemented, etc.
4 replies →