Comment by suriya-ganesh

This is a problem that occurs across game development to ML datasets.

We built oxen to solve this problem https://github.com/Oxen-AI/Oxen (I work at Oxen.ai)

Source control for large data. Currently our biggest repository is 17 TB. would love for you to try it out. It's open source, so you can self host as well.