Comment by walrus01

9 hours ago

I assume you mean putting only the 32B active parameters on the GPU, and the rest on a bunch of regular server DRAM like on a 768GB to 1024GB RAM server?

Because Kimi K2.6 in Q4 is about 584GB GGUF size on disk and will use slightly more than that in RAM, Q8 is 595GB.