Comment by unsnap_biceps
2 days ago
Isn't that true only if the batches are different? If you run exactly the same batch, you're back to a deterministic result.
If I had a black box api, just because you don't know how it's calculated doesn't mean that it's non-deterministic. It's the underlaying algorithm that determines that and a LLM is deterministic.
Providers never run same batches because they mix requests between different clients, otherwise GPUs are gonna be severely underutilized.
It’s inherently non deterministic because it reflects the reality of having different requests coming to the servers at the same time. And I don’t believe there are any realistic workarounds if you want to keep costs reasonable.
Edit: there might be workarounds if matmul algorithms will give stronger guarantees then they are today (invariance on rows/columns swap). Not an expert to say how feasible it is, especially in quantized scenario.