Comment by zargon
9 hours ago
> emerging practice of using Q4 quants and Q8 KV cache for local inference
That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.
I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.
Gemma's QAT is surprisingly good (although Gemma isn't that great to begin with).
IME: Gemma is not great for programming, but it is fantastic at following directions compared to anything else in its size class.