Comment by dannyw

They clearly list out the quants in Table 1.

Most of the following tables show relative speedups; not tok/s numbers.

I agree no home AI users should really use bf16^; but I see two defensible justifications: (1) showing that their new hardware's speedups scale to full precision too [crucially; NVIDIA is focusing more and more on lower quants], and (2) trying to sell to the crowd running IQ3XXS or something on 16GB.

^q8 ~= bf16 is a good general rule of thumb. And friends don't let friends quantize KV cache.