Comment by rhdunn
10 hours ago
The A3B part of that model means that each "expert" in the mixture of experts (MOE) model use 3B parameters. At 4-bit quant, you can fit 2 parameters per byte. That gives you 1.5GB per expert (plus a bit for overhead), which makes it usable on an 8GB card.
It's all about trade-offs: how much of the model you want to load into VRAM; how much VRAM you want to reserve for context; how much quantization you/the model can tolerate; how many tokens/second you can tolerate; whether you want to use the VRAM for anything else (graphics, Text-to-Speech, etc.); etc.
A3B refers to 3 billion parameters active during a forward pass, not the size of an individual expert.
The 3 billion includes all active experts, and then other shared weights, attention components, embeddings.
In the qwen 3.5 35b a3b series, each expert is ~3 million parameters. There are 40 layers which each have 1 shared expert and 8 routed experts. Roughly 1.1B of the 3B active are experts.
Yes, this is all good knowledge in theory for testing local ai. There are some more technical stuffs to consider that are happening in the background that is well outside my expertise to explain. I have been testing every MoE model I can fit on my system every year to find out which is the best for me.
Last year, it would have been impossible to run a 35B A3B model on old Pascual gpu and make it somehow usable. But for some reason now it just works even with a better finetuned version. So clearly there is more to it that just having it theoretically fit the system. Look for example how Strata works on certain models and systems compared to its llamacpp counterpart.