Comment by AntiRush
2 hours ago
I've been working on support for this model in ds4 on the RTX 6000 pro - it's been really great for my use cases. The ds4 q4 quant performs a lot better than other similar sizes that I've seen.
Using the Q4 quant on an RTX 6000 Pro Workstation Edition at 450 watts:
Code: prefill 1,251 tok/s decode 255.26 tok/s
Prose: prefill 1,251 tok/s decode 198.78 tok/s
Most important for me, I can run 4 concurrent streams at 400+ tok/s.
DS4 is an odd model. I have it working on way too many GPUs and yet for many tasks Qwen 3.8 will do much better. It also tends to loop, which is super annoying.
GLM5.3 runs on similar hardware and is much better so if you're going to burn cycles and brain power on this maybe look at GLM5.3 as a comparison as well?
Other than that, when you're done with that card...
ds4 is the inference engine not the model.