Comment by skolos
18 hours ago
On many models that I tested in past context quantization had very bad effect on model performance. However qwen3.8 27b is different.
I'm now running NVFP4 quantized both weight and cache on my RTX5090 and getting excellent results: 264k cache allocated for pool, 10k tok/s prompt processing, 200 tok/s generation for single stream, or 801 tok/s generation for 8 concurrent streams. Also have about 2Gb vram left for use of OS.
my coding agents regularly reach 200k context used without noticeable degradation.
P.S. I used setup from: https://github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
I’m running 27B on a 5090 as well, and the results have been really strong. It does almost as well as, and sometimes better than, a 121gb DS4 model running on an M5 Max 128gb. 27B also flies on the 5090, and at medium think it returns results many times faster than my DS4 setup (the default xhigh is basically broken, though).
For the kinds of things I use a local model for (legal document review), it’s just spectacular. It also has good vision support. I’ve been using 27B more and more over DS4.
Those are really nice numbers. With that t/s, no network latency or queueing it must feel much snappier than cloud models.
it is, unless it set to xhigh - it really likes generating tons of tokens for its thinking. unfortunately, for decently reliable coding results you want it on xhigh ...