Comment by explainit2me
11 hours ago
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
11 hours ago
So this compression is only pertinent to the LLM file format? In memory it'd have to be expanded into the 1.58-bit form - 5 trits per byte.
It also means you can read them faster, more parameters per second during an inference which tends to be memory bandwidth limited on most systems. Thus faster inferences