Comment by bityard
4 hours ago
I've read that page before and although it all certainly sounds very impressive, I'm not an AI researcher. What's the actual goal of dynamic quantization? Does it make the model more accurate? Faster? Smaller?
4 hours ago
I've read that page before and although it all certainly sounds very impressive, I'm not an AI researcher. What's the actual goal of dynamic quantization? Does it make the model more accurate? Faster? Smaller?
More accurate and smaller.
quantization = process to make the model smaller (lossy)
dynamic = being smarter about the information loss, so less information is lost