Comment by vincston
3 hours ago
Can you elaborate? Is it because the models are inherently not deterministic? But then, will it not get better with time? I mean arent we just at the beginning of the research here?
3 hours ago
Can you elaborate? Is it because the models are inherently not deterministic? But then, will it not get better with time? I mean arent we just at the beginning of the research here?
> Is it because the models are inherently not deterministic
This is the main reason (and yes, many modern compilers and CPUs carry some non-determinism which actually quite well explained and specified).
> But then, will it not get better with time?
No, it won't. Bacuse that is that is the actual literal limitation/feature of LLMs.
> I mean arent we just at the beginning of the research here?
In general? Yes. With LLMs? We can reasonably say that they will never be deterministic.
There is a way to get a non-determenistic output: first question on a temp 0 local model on a completely new session will give you the same answer. The second answer in that same session will already be different on every session (even if it's the same question).