Comment by colechristensen

You may or may not be familiar with the content defined chunking family of algorithms like FastCDC.

https://joshleeb.com/posts/chunking.html

https://www.usenix.org/conference/atc16/technical-sessions/p...

On the side I'm working on an extension to git-lfs to use fastcdc for both storage and transmission of artifacts to drastically reduce size and make git-lfs more practical for more things.