Comment by vforno

It’s a good question.

In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps.

Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).