Comment by liuliu

3 days ago

FWIW, single M5 Ultra with 256GiB runs DeepSeek 4.1 Flash 0910 just fine (~3200 tok/s prefill, ~45 tok/s). M5 Max w/ 128GiB runs GLM 5.3 Flash also fine.

I also benchmarked both on quality benchmarks (MathArena Apex / GPQA Diamond etc) to make sure the quality is what you get from the official numbers.

7 comments

liuliu

Reply
TomGarden  2 days ago

GLM 5.3 Flash on 128GB Ram would have to be 2-bit right?

I'm no expert but it seems the consensus is you get a suboptimal experience under 4-bit. Please correct me if I'm mistaken here

  • liuliu  2 days ago

    MoE, especially these Chinese ones trained on low-bits already is actually quite quantizable to lower bits, unlike smaller / dense models.