Comment by spockz
7 days ago
You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.
7 days ago
You can run smaller models on smaller compute hardware and split the compute. For large models you need to be able to fit the whole model in memory to get any decent throughput.
Ah interesting, I didn't realize MoE doesn't need to all run in the same place.