Comment by adrian_b

This is partially mitigated by the fact that all the formats that quantize to 4 bits, or other such low values, partition the weights into small blocks and they also keep scale factors for each block of 4-bit weights.

This works well when the dynamic range of the weights does not vary much within a block, but it fails when closely located weights have very different magnitudes.

NVFP4 is more accurate than other 4-bit formats, because it stores more scale factors, i.e. 1 FP8 scale factor for each block of 16 4-bit weights, plus 1 FP32 scale factor for each tensor.

MXFP4 uses blocks of 32 values, and the common scale factors are only powers of two (which provides a higher dynamic range than FP8, but a coarser resolution).