Comment by seanmcdirmid

I wouldn’t know since I haven’t found a model to test it on, but in my experience with smaller MoE models (35b A3b), they are the only choice for my M3 Max because prefill times are reasonable (I don’t even need to quant my KV cache, which would mess up MTP anyways). The only reason dense models (3.8 27b) are unusable for me are the high prefill times (mitigated if prefix caching is working correctly).