Comment by troupo

> Is it because the models are inherently not deterministic

This is the main reason (and yes, many modern compilers and CPUs carry some non-determinism which actually quite well explained and specified).

> But then, will it not get better with time?

No, it won't. Bacuse that is that is the actual literal limitation/feature of LLMs.

> I mean arent we just at the beginning of the research here?

In general? Yes. With LLMs? We can reasonably say that they will never be deterministic.

There is a way to get a non-determenistic output: first question on a temp 0 local model on a completely new session will give you the same answer. The second answer in that same session will already be different on every session (even if it's the same question).