Comment by anon373839

HF uses chunk-level deduplication that greatly reduces the amount of physical storage needed for a given amount of logical storage.

https://huggingface.co/docs/xet/en/deduplication