Comment by Havoc
1 day ago
I'm guessing by lossless they mean something other than what the word usually means in compression context?
>achieving near information-optimal compression without any loss of precision
So perhaps more lossless as in didn't lose perplexity/benchmarks?
In my mind lossless is precisely zero bits lost along the way.
The first sentence of the introduction ends with "we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model" so yes it's lossless.
information-optimal compression is "the theoretical minimum number of bits needed to represent data without losing any information, based on the data's entropy", so I think they mean the same thing you do
Yeah, they’re saying that this compression is almost as good as is theoretically possible without losing any information.
A good example that information, i.e. bits, are only meaningful with respect to an end. If you don't know what the bits in a float will be used to, you can't throw them away, but if the floats are in a function, and you know that what some bits are can't affect the output of the function regardless of input, then you can throw those bits away and still have a lossless compression of the function.
Think Morse code, where frequently used letters have shorter codes than less frequent ones. This ensures zero loss of information.
The part you quote is a few sentences past the sentence that says "preserving outputs that are bit-for-bit identical to the original model".