Comment by PcChip
7 days ago
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
7 days ago
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.
Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.
It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
Thanks. This relates to some questions I've got. I was playing around with the previous generation smaller models and found that i wasn't getting any speedups from the T1 and T2 binary/ternary models compared to standard Q4 quants of straight qwen3.6 models. I was wondering whether unpacking of the ternary encoding was impacting inference speed?
If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.
packing multiple trits together
e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing
It's impressive how close to optimal this is.
You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59).
At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.
> never heard of a bit ever having more than two possible values
It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer.
The way they do it is packing like the other comment says.
Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.