Comment by giantrobot

Token generation is nominally bandwidth limited. Prefill/prompt processing is nominally compute limited.

For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.