Comment by eightysixfour
1 day ago
The models themselves are mathematically deterministic. We add randomness during the sampling phase, which you can turn off when running the models locally.
The SaaS APIs are sometimes nondeterministic due to caching strategies and load balancing between experts on MoE models. However, if you took that model and executed it in single user environment, it could also be done deterministically.
> However, if you took that model and executed it in single user environment,
Again, are those environments in the room with us?
In the context of the article, is the model executed in such an environment? Do we even know anything about the environment, randomness, sampling and anything in between or have any control over it (see e.g https://news.ycombinator.com/item?id=44528930)?
It's very poor communication. They absolutely do not have to be non-deterministic.
The output of all these systems used by people not through API is non-deterministic.
1 reply →