Comment by pixl97

23 days ago

In this workload, it happened to be, in a lot of workloads it is not at all. This is quite likely because over time media sizes have increased greatly. Microsoft research on operating systems shows that contiguous chunks at 64MB don't really show any IOPS penalty in practical usage scenarios.

I work on workloads that work with thousands to millions of small files. 1k to 1MB. I can promise you that the IO effect is massive, with shortcuts like working directly on archive files/zips where they are read and written at once saves massively on hardware costs as IOPS tend to be expensive. This is with fast NVMe disks, no spinning platters involved. Also small files massively increase write amplification on SSDs.

There probably is significant savings in deduping 'tiny' files, but the moment you get past a particular threshold it drops off, and the vast majority of game assets have increased over that size.