Comment by bensyverson
8 hours ago
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
8 hours ago
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
And is this using conventional model loading (all in VRAM), or are you streaming it in some way?
in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.