Comment by zozbot234
3 hours ago
> ...Back when 4k movies needed expensive hardware, no one was saying they could play 4k on a home system, then later mentioning they actually scaled down the resolution to make it possible. ...
int4 quantization is the original release in this case; it's not been quantized after the fact. It's a bit of a nuisance when running on hardware that doesn't natively support the format (might waste some fraction of memory throughput on padding, specifically on NPU hw that can't do the unpacking on its own) but no one here is reducing quality to make the model fit.
Good point thanks for the clarification.
The broader point remains though which is, “you can run this model as home…” when actually the caveats are potentially substantial.
It would be so incredibly slow…