Comment by om8
15 hours ago
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
15 hours ago
Ternary quantization does not make any sense. Vector quantization and trellis based methods are better in this region for PTQ.
PTQ and vector quantization aren’t used for this because part of the point of ternary LLMs is to make them faster. In a ternary LLM every weight is an add, subtract, or no-op so it is fast on CPU.
If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
And storing it in memory. Memory is expensive.
which is important though since sending it across the wire over and over and over is actually the main bottleneck.
Wire typically means internet connection, and it’s hardly the bottleneck
3 replies →
> If you’re just using a code book to reconstruct a f16 model the only savings you can get are in sending it over the wire.
That’s why you need to use efficient gemm kernels like FLUTE for inference. They are ~as good as what you can do with ternary quantization.
If you want sub-2 bit llm, get one that’s already trained in higher precision, and compress it with something like YAQA/QTIP with finetuning or PV-tuning + AQLM/HIGGS