Comment by kennywinker

Yeah 16gb. For longer context, a 3bit quant is needed. Yes it’s tight on a 16gb card - can’t wait for the bubble to pop so hardware prices fall.

But I don’t quite follow you - how does a more aggressive quant slow it down? Less bits per token means faster inference not slower.