Comment by jacquesm

Specifically:

  5257408 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf
  49215492960 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00002-of-00005.gguf
  49700372160 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00003-of-00005.gguf
  49466495968 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00004-of-00005.gguf
  13481997024 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00005-of-00005.gguf

  49842093280 GLM-5.3-Flash-Q3_XL-3.86bpw-00001-of-00004.gguf
  49644428576 GLM-5.3-Flash-Q3_XL-3.86bpw-00002-of-00004.gguf
  49653319264 GLM-5.3-Flash-Q3_XL-3.86bpw-00003-of-00004.gguf
  5694702080  GLM-5.3-Flash-Q3_XL-3.86bpw-00004-of-00004.gguf

  20437336576 qwen3_8_27b.ninfer

The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.

Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.

More details here:

https://jacquesmattheij.com/local-inference-with-large-langu...