Comment by boredatoms
20 hours ago
For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain
20 hours ago
For sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain
Yeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.
NVFP4 would buy them a huge increase in capacity but I think it would be noticeable.
Why doesn't someone just try to measure this next time!?
1 reply →
Some people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.
Could this explain Opus?