Comment by Dwedit
3 days ago
Given the same text input, same hardware type (inference on CPU vs on GPU), and same RNG seed, you will get the same exact output. That is deterministic.
That was from when I was testing out someone's claim that koboldcpp got less smart in a newer release, but I was reproducing the exact same output across different versions of the program.
I thought modern CPU has TRNG ?
For LLMs, you're picking a result token from a list of tokens and their probabilities. It's not a situation like cryptography where you need secure random numbers. Just traditional and simple RNGs (seeded using the system clock) are useful enough for this task.
For the koboldcpp/llama.cpp example, there is a setting where you can change the initial RNG value. It says "seed" in sampler settings. Set it to any constant besides 0 and you get consistent behavior.