Comment by gpm
7 months ago
> or even a single socket Epyc system with >1TB of RAM
How many tokens/second would this likely achieve?
7 months ago
> or even a single socket Epyc system with >1TB of RAM
How many tokens/second would this likely achieve?
KTransformers now supports Kimi K2 for MoE offloading
They claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory.
around 1 by the time you try to do anything useful with it (>10000 tokens)
1