Comment by schubidubiduba

3 hours ago

I am running that on a single 6000 96GB with 4-bit quants for both weights and PLE table. Needs just 32GB RAM and fits snugly into the 96GB VRAM with KV cache equalling ~300k context tokens. Not sure if I quantized the KV

0 comments