Comment by mmastrac
1 day ago
Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.
I think I was just following along with the previous post about running Gemma on a Xeon. Next I’m going to see which model can give the highest tokens/sec
Such a system is RAM bandwidth limited and not compute limited Switching to q4 from q8 would decrease the amount of data needing to be loaded by half. The token generation rate would nearly double. But generally if you can do q6 or q8 and you have enough RAM you really should. Even if it's slower.
Token generation is nominally bandwidth limited. Prefill/prompt processing is nominally compute limited.
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.