Comment by alentred
17 hours ago
I would be very interested in a similar benchmark for *KV cache* quantizations.
I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.
I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.
There was a study specifically related to Qwen3.8 27B that showed that kv cache quantization has almost no impact on this model all the way to q4:
https://arxiv.org/html/2609.04098
On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.
I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.
my coding agents regularly reach 200k context used without noticeable degradation.
P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
I’m running 27B on a 5090 as well, and the results have been really strong. It does almost as well as, and sometimes better than, a 121gb DS4 model running on an M5 Max 128gb. 27B also flies on the 5090, and at medium think it returns results many times faster than my DS4 setup (the default xhigh is basically broken, though).
For the kinds of things I use a local model for (legal document review), it’s just spectacular. It also has good vision support. I’ve been using 27B more and more over DS4.
Those are really nice numbers. With that t/s, no network latency or queueing it must feel much snappier than cloud models.
it is, unless it set to xhigh - it really likes generating tons of tokens for its thinking. unfortunately, for decently reliable coding results you want it on xhigh ...
You don't have to quantize all layers and all dimensions uniformly, FWIW