Comment by tarruda
8 hours ago
> It also includes additional 196B Engram memory which you can put on an SSD. I think
You can put Qwen 3.8 Flash Next engram on SSD, but prompt processing takes a good hit. On my mac studio, I get 300 pp and 33 tg with SSD offload, versus 550/40 with everything in RAM.
I will be very happy if 300 pp is achievable with this model though.
The engram stuff is great because RAM is often still cheaper (or at least expandable). My company does currently look into buying some hardware as we handle confidential data and code.
Qwen 3.8 Flash is viable on two Nvidia 6000 96GB with a wood quant because you can put the 50GB Engram into RAM and the hit should be below 10% performance. At least that is what I have seen so far. Correct me if I'm wrong.
I am running that on a single 6000 96GB with 4-bit quants for both weights and PLE table. Needs just 32GB RAM and fits snugly into the 96GB VRAM with KV cache equalling ~300k context tokens. Not sure if I quantized the KV
You can warm cache regularly used engram/n-gram if you're willing to merge PRs into a personal branch and build it yourself. I was trying this with qwen 3.8 flash next and the n-gram to get it to fit on my very average gaming desktop (it worked)