Comment by Alifatisk

I think it's because of a combination between the MoE model architecture and the inference done in large batches and run in parallel