Comment by nemo1618

It warms my heart to see so many people in the comments clarifying that LLMs can indeed be deterministic. This was not always the case!

I hope that we will soon see deterministic inference offered by frontier labs (presumably at a premium rate). In the meantime, you can get deterministic behavior from local models using the right vLLM flags. I've described some of the cool applications of determinism here: https://lukechampine.com/repligraphs