Comment by behnamoh
14 days ago
does the entire model have to be loaded in VRAM? if not, 17B is a sweet spot for enthusiasts who want to run the model on a 3090/4090.
14 days ago
does the entire model have to be loaded in VRAM? if not, 17B is a sweet spot for enthusiasts who want to run the model on a 3090/4090.
Yes. MoE models tipically use a different set of experts at each token. So while the "compute" is similar to a dense model equal to the "active" parameters, the VRAM requirements are larger. You could technically run inference & swap the models around, but the latency would be pretty horrendous.
I think prompt processing also needs all the weights.
Oh for perf reasons you’ll want it all in vram or unified memory. This isn’t a great local model for 99% of people.
I’m more interested in playing around with quality given the fairly unique “breadth” play.
And servers running this should be very fast and cheap.