Comment by r3trohack3r

9 hours ago

Tangentially related, I applied a similar approach to compress the npm registry by over 90% on disk a few years back. Since most versions of a package are similar, you can delta encode them first and then compress them. The deltas are small and compress well as a collection with the original source files.

For another use case, prior to compressing, I’ve applied a rolling hash to deterministically split the file. Then compressed the chunks and stored them in a CID filesystem. The result is that files that are largely similar share compressed chunks.

There are a lot of things we can do to be substantially more efficient with the computers we have, but engineers often cost more than hardware. With recent supply chain constraints that calculus is changing!

6 comments

r3trohack3r

Reply
oefrha  2 hours ago

NPM packages doesn’t take up that much disk space for me with standard pnpm deduplication. The much bigger offender for me is Rust target dir: when people talk about vibe coding in Rust for performance, what they don’t mention (at least I’ve hardly ever seen it mentioned) is every trivial little tool pushes 1GB on disk and anything slightly nontrivial easily racks up multi-GB. Which hurts when you have lots of vibed small tools. I wonder if anyone bothered to look into this problem.

  • jmtulloss  2 hours ago

    I think the parent was referring to hosting the entire NPM registry, not having a project that uses NPM. In that case there's substantial duplication from version to version.

a_t48  8 hours ago

I've done similar things for large container images. My format allows for using FastCDC to chunk files, but there's a tradeoff between number of shared chunks and between number of HTTP requests. I keep it turned off by default.

Twirrim  6 hours ago

I'd be curious whether block level de-duping would add value too in their case. You effectively achieved that to some degree with storing the deltas.