Comment by dannyw

We're looking at a MoE with 50B active params, each inference pass only requires the compute of a 50B dense model.