Comment by nok22kon 1 day ago you can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B. 2 comments nok22kon Reply zingar 1 day ago Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error? nok22kon 20 hours ago paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bithttps://arxiv.org/abs/2212.09720
zingar 1 day ago Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error? nok22kon 20 hours ago paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bithttps://arxiv.org/abs/2212.09720
nok22kon 20 hours ago paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bithttps://arxiv.org/abs/2212.09720
Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?
paper is a bit old, but matches current empirical recommandation: a good starting point is the biggest model you can fit at 4 bit
https://arxiv.org/abs/2212.09720