Comment by andy99
15 hours ago
> Qwen3.8-Flash-Next features a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token.
Didn’t see this mentioned yet. I wonder what this means for the effective size. It’s evidently ~176B paramètres, but how does that get quantized. A 4-bit quant under 100GB seems unlikely, I’m suspecting this won’t run in 128GB unified memory
In principle I like the idea of trading more memory for compute though, even if there’s a memory shortage right now
It is 125B A6B. vLLM is already out with support, ngrams can be offloaded to RAM so you only need ~96GB VRAM for nvfp4 w/ full context.
Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.
The N-gram parameters can be fetched from SSD, with maybe the hottest ones staying in memory.
I have this working on a branch of my https://github.com/rdaum/eider (for DGX Spark)
nVME paging the n-gram table (in BF16 for now).
Still working at it. Prefill sucks still but decode is about 12 tok/sec and the model weights fit nicely in the 128GB Spark memory in nvfp4 quant while paging the ngram stuff from disk.
(EDIT: merged to main. 80tok/sec prefill, 12 tok/sec decode, ~80GiB resident, the rest paged)
People in my server are running it on Strix Halo 128GB using RoCmFP4 and reporting 35tok/s, without much optimization, with proper MTP, better kernel, expecting about 50-60tok/s.
How do they like it, compared with 3.8 and DS4?
Gonna have to wait a few days to see what the wizards of the HF community come up with…
They are already working on it.
https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF https://unsloth.ai/docs/models/qwen3.8-next
> You will need at least 75 GB of RAM or unified memory to run the model. Its smallest 1-bit quantized version is larger than usual because of the model’s architecture so 1-bit isn't really 1-bit at all. However, this also means the quantization is less aggressive, allowing the model to retain more of its original accuracy than more heavily quantized models.
Lots of RAM required even for the 1-bit, which is already downloadable. Interested to see how well this one works compared to Ornith1.5-35B-A3B I've been running (and quite happy about).
Edit: but llama-cpp does not yet support it.
The PR branch does seem to work, I'm planning to move almost all of my Qwen using workload over to it tonight.
I have nvfp4 quant fitting fine in 128GB on DGX Spark, but with paging (from nVME) of the n-gram table. Resident ~80GiB for weights & context.
On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later)
Still actively working on this. Prefill currently sucks. Will merge to main by end of day.
EDIT: This has now landed on main. Still haven't done MTP speculative decoding boost, but:
80tok/sec prefill, 12 tok/sec decode. ~90GiB or so resident. n-grams paged from disk.