Comment by esafak
2 days ago
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
2 days ago
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
> You have to look at the size of each expert
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
Yeah, "experts" is a ML/research word for this (MoE was first published in 1991; and has been around for a long time, it even predates deep learning). it's not the everyday/colloquial meaning of 'expert'.