Comment by Aurornis

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.

This prediction alone isn’t useful at all without a bound on speed and maybe quantization.

You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.

> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.

You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.

You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.

Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.