Comment by jamesblonde
2 days ago
For the full weights, unoptimized on vLLM with 2 Nvidia 6000 RTX 48GBs connected by NVLink, i only get 14 tokens/sec with open-code. For batched operations, it climbs to 55 tokens/sec.
For FP8, on a single Nvidia 6000 RTX 48GB, i get 13 tokens/sec on a single GPU and 46 tokens/sec batched.
On 2x RTX A6000 non-nvlink connected but communicating across the CPU, with llama cpp I get ~60 tok/s for Qwen3.8-27B-UD-Q8_K_XL without any batching.