Comment by TomGarden
2 days ago
GLM 5.3 Flash on 128GB Ram would have to be 2-bit right?
I'm no expert but it seems the consensus is you get a suboptimal experience under 4-bit. Please correct me if I'm mistaken here
2 days ago
GLM 5.3 Flash on 128GB Ram would have to be 2-bit right?
I'm no expert but it seems the consensus is you get a suboptimal experience under 4-bit. Please correct me if I'm mistaken here
MoE, especially these Chinese ones trained on low-bits already is actually quite quantizable to lower bits, unlike smaller / dense models.
Also Expert MoE offload (to SSD), which still doesn’t let me (64GB) run a flagship but if you have 96GB you can.
The problem with SSD offload is the prefill time suffers, which is not really obvious.
3 replies →