Comment by binary132
9 hours ago
This is not accurate, I get like 80t/s on Q3.8 27b UD Q4_K_XL on my 4090 with 3 drafts and about 110K context window, sim shows mid 20s
9 hours ago
This is not accurate, I get like 80t/s on Q3.8 27b UD Q4_K_XL on my 4090 with 3 drafts and about 110K context window, sim shows mid 20s
I get about 50 t/s with the same model on a 3090, with about 100K context. I usually set harnesses to 90k.
It would be neat to also include NVFP4 variants for the Blackwell hardware.