Comment by v9v
2 hours ago
It still works the same way other LLMs do, by outputting the probability distribution over the possible completions (The weather is ... (sunny (50%), cloudy (50%))). Then the next token is sampled from this probability distribution (in our example the next word could be "sunny" or "cloudy" equally likely), which can result in different outputs every run.
Could the model or algorithm be changed to make it deterministic somehow? It could help a lot if there were reproduceable outputs from deterministic baked-in silicon.