I wouldn’t know since I haven’t found a model to test it on, but in my experience with smaller MoE models (35b A3b), they are the only choice for my M3 Max because prefill times are reasonable (I don’t even need to quant my KV cache, which would mess up MTP anyways). The only reason dense models (3.8 27b) are unusable for me are the high prefill times (mitigated if prefix caching is working correctly).
MoE weights are much friendlier to low-bit quants than dense weights, because only a few experts are active per token so the per-expert error stays local. But the trap people fall into is that once weights are down to 2-3 bits, the KV cache and activation buffer at long context become the dominant term — further weight quantization stops moving the number. Memory-planning for MoE is really a context-length problem wearing a quantization costume.
I wouldn’t know since I haven’t found a model to test it on, but in my experience with smaller MoE models (35b A3b), they are the only choice for my M3 Max because prefill times are reasonable (I don’t even need to quant my KV cache, which would mess up MTP anyways). The only reason dense models (3.8 27b) are unusable for me are the high prefill times (mitigated if prefix caching is working correctly).
MoE weights are much friendlier to low-bit quants than dense weights, because only a few experts are active per token so the per-expert error stays local. But the trap people fall into is that once weights are down to 2-3 bits, the KV cache and activation buffer at long context become the dominant term — further weight quantization stops moving the number. Memory-planning for MoE is really a context-length problem wearing a quantization costume.
I think that is true for GLM-5.3 to some extents, this is not true for DeepSeek 4-series models as they compress KV aggressively.