Comment by zozbot234

For sparse MoE models, the single expert layers that the inference gets sampled from are actually quite small - single-digit megabytes or so.