Comment by petu

You need VRAM for the whole thing for optimal performance. Activation is chosen "randomly" for each token. PCIe becomes bottleneck, so much that just doing computation on CPU is likely faster.

But given it's only 6B, out of which only ~2.4B seem to be actually routed ("selected at random per token"), you could get reasonable performance with experts on CPU (still haven't tested, but 20-30 for dual channel DDR5 and 4 bpw quant).