Comment by mixermachine
3 hours ago
The engram stuff is great because RAM is often still cheaper (or at least expandable). My company does currently look into buying some hardware as we handle confidential data and code.
Qwen 3.8 Flash is viable on two Nvidia 6000 96GB with a wood quant because you can put the 50GB Engram into RAM and the hit should be below 10% performance. At least that is what I have seen so far. Correct me if I'm wrong.
I am running that on a single 6000 96GB with 4-bit quants for both weights and PLE table. Needs just 32GB RAM and fits snugly into the 96GB VRAM with KV cache equalling ~300k context tokens. Not sure if I quantized the KV