Comment by Havoc

10 hours ago

The bonsai quants are not that awesome. Yeah they're fast and they retain the "27B" but in practice the perform similarly

5GB of bonsai weights end up being similar to 5GB of QAT weights even when one is derived from a 27B and the other from an 8B