Comment by _0ffh
13 days ago
As a rule of thumb, the bigger the model is, the more graciously it degrades under quantisation. So you may assume performance loss for a 8B model would be lower than for a 3B model. (I know that doesn't make up for missing numbers in link, just fyi.)
No comments yet
Contribute on Hacker News ↗