Comment by andai
2 days ago
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
Or, GLM 5.2 simply had more time in the RL oven.
Deepseek V4 Flash, the 284B model, is roughly equivalent to launch GLM 5, the 744B [sic] model.
After MoE entered the mix, raw parameter count is less useful a measure.
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
> You have to look at the size of each expert
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
Yeah, "experts" is a ML/research word for this (MoE was first published in 1991; and has been around for a long time, it even predates deep learning). it's not the everyday/colloquial meaning of 'expert'.
It's almost like they priced models based on their performance or something...