Comment by jacquesm

10 hours ago

I have all three and you're not going to convince me of that.

Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3 and faster than that if you get the speculative engine set up properly. That means you can iterate very rapidly on the same problem and for many problems this is the way to solve them. DS4 and GLM 5.3 shine when Qwen would take too many turns to solve something, they simply have more world knowledge and seem to home in on possible solutions faster in #tg but slower in wall clock time. If a particular problem is within striking range of all three then Qwen will usually be the fastest solver by far.

It also depends on how good your prompt is, whether you are setting up points for guidance or whether you expect everything to be solved in a single shot.

2 comments

jacquesm

msdz 8 hours ago

> Side-by-side Qwen 3.8 27B is ~6x as fast as DS4 and GLM 5.3

Wow, that sounds like a lot! Do you use/mean the regular models or flash variants here?

jacquesm 8 hours ago
Specifically:
5257408 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf 49215492960 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00002-of-00005.gguf 49700372160 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00003-of-00005.gguf 49466495968 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00004-of-00005.gguf 13481997024 DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00005-of-00005.gguf 49842093280 GLM-5.3-Flash-Q3_XL-3.86bpw-00001-of-00004.gguf 49644428576 GLM-5.3-Flash-Q3_XL-3.86bpw-00002-of-00004.gguf 49653319264 GLM-5.3-Flash-Q3_XL-3.86bpw-00003-of-00004.gguf 5694702080 GLM-5.3-Flash-Q3_XL-3.86bpw-00004-of-00004.gguf 20437336576 qwen3_8_27b.ninfer
The GLM Quant is ~4 bpw, the DS4 is a Q8 and ninfer one is an NVFP4/BF16 I think. I run the ninfer one with ninfer-4090 on a hacked card with 48GB of VRAM. Max context for that one is 256K, DS4 has a 1M context and for GLM 5.3 it is 300K. The big models use all 8 GPUs.
Qwen benefits in two ways: all of the VRAM is on a single GPU so there is no communications other than generated tokens and new activations, that gives pretty good utilization for the card, the second is of course simply the fact that it is a much smaller model so there is less to compute. The 4090 is still power limited so it could do more but it's loud (and hot) enough as it is.
More details here:
https://jacquesmattheij.com/local-inference-with-large-langu...