Comment by vismit2000
1 day ago
This remains one of the best explanations on the topic: https://fabiensanglard.net/floating_point_visually_explained... Saw this when I just started using HN and such posts only inspired me to stick to it: https://news.ycombinator.com/item?id=29368529
Maybe I am too mathematically enclined, but this was not easy to understand.
The ELI5 explanation of floating point: they approximately give you the same accuracy (in terms of bits) independently of the scale. Whether your number if much below 1, around 1, or much above 1, you can expect to have as much precision in the leading bits.
This is the key property, but internalizing it is difficult.
I like "between each power of 2, there are the same number of numbers."
So between 1/2 and 1 there are the same number of numbers as between 1024 and 2048. If you have 1024 numbers between each power of 2, then each interval is 1/2048 in the first case and 1 in the second case.
I reality there are usually:
bfloat16: 128 numbers between each power of 2
float16: 1024 numbers between each power of 2
float32: 2*23 numbers (~8 million) between each power of 2
float64: 2*52 numbers (~4.5 quadrillion) between each power of 2
Or, say, you can write any number you want, but it has to be a whole number from 0 to 9, and you can only make the number bigger or smaller by moving the decimal point, and you can only move the decimal point up to 10 spaces. And you can add or remove a negative sign in front.
and makes so much sense in the context of this recent blog from TM's researchers
https://news.ycombinator.com/item?id=45200925
I've never seen this topic so well explained, thank you for sharing!
From just scanning through the article quickly, I don't see there the mantissa similarly easily explained, but there is a very intuitive way to think of it. Because the mantissa (like everything else) is encoded in binary, the first explicit (because there's implicit 1. at the beginning) digit of it means either 0/2 or 1/2 (just like in decimal the first digit after the dot means either 0/10 or 1/10 or 2/10...), the next digit is (0/2² = 0/4) or 1/4, third digit is 0/8 or 1/8 etc. You can visualize this by starting at the beginning of the "window", and then you divide the window into 2 halves: now the first digit of the mantissa tells you if you stay at the beginning of the first half, or move to the beginning of the 2nd half. Now whatever half you picked, you divide it into 2 halves again, and use the next bit of mantissa to tell you if you should advance to the next half. So you just keep subdividing, and so the more bits in the mantissa you have, the more you can subdivide the window, and if the exponent (after applying bias) is equal exactly the number of explicit bits in the mantissa, the smallest subdivision cell has length equal exactly 1. Incrementing exponent by 1 will now double the window size, and without additional subdivision each cell has length equal exactly 2 meaning each next float number now increments by 2.
(keep in mind there is a subnormal range where there's implicit 0. at the beginning of the mantissa instead)
To reiterate, increasing the exponent by 1 doubles the window size, so the exponent describes how many times the window size was doubled while the number of bits of mantissa describes how many times you can do the reverse and "half" it, hence the exponent to mantissa bits relation.