Comment by torginus

Afaik, for a MoE model, total size doesn't matter that much for how heavy it is to run, the size and number of active experts at any given time does. Of course, you still have to store the whole model in fast memory, so there's that, but the reason these things are getting this large is because it doesn't really affect runtime that much.