Comment by Dwedit

For LLMs, you're picking a result token from a list of tokens and their probabilities. It's not a situation like cryptography where you need secure random numbers. Just traditional and simple RNGs (seeded using the system clock) are useful enough for this task.

For the koboldcpp/llama.cpp example, there is a setting where you can change the initial RNG value. It says "seed" in sampler settings. Set it to any constant besides 0 and you get consistent behavior.