Comment by wyrdcurt
2 hours ago
Except that even the exact same model won't output the exact same results, that's a fundamental aspect of how LLMs work. They're probabilistic/stochastic, not deterministic.
2 hours ago
Except that even the exact same model won't output the exact same results, that's a fundamental aspect of how LLMs work. They're probabilistic/stochastic, not deterministic.
Models are weights for matrix operations, they are determenistics.
Achieving determinism with LLMs and other neural network models is actually a hard problem that people spend a lot of time on, when they need that. It doesn’t happen by accident.
Issues include accumulated floating point errors happening in different orders due to distributed and parallel computation, CUDA kernels that deliberately sacrifice determinism for speed, and several other such issues.
They are weights for matrix operations, so in principle you'd think they would be deterministic. In practice it's more complicated than that.
Not to be snarky or dismissive, I mean this genuinely: ask an LLM about it. I currently have a headache so I'm not up to explaining the technical details, but they are interesting and worth reading about.