Comment by johndough

Update: Looks like the model is larger after all (1561.44 GB). Only the MoE weights are MXFP4, while the other weights are BF16 (and a few FP32).

* Sparse Experts: 1481.4 GB

* Dense Experts: 1.9 GB

* Self-Attention: 72.4 GB

* LLM Head: 2.4 GB

* Embeddings: 2.4 GB

* Vision Encoder: 0.35 GB (surprisingly small)

plus some miscellaneous parameters.

Most importantly, we now know that the model has 104B active parameters, which is quite a lot and will make it difficult to self-host efficiently.