Comment by a_t48
1 day ago
The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)
1 day ago
The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)
xet on hugging face does seem to not have such bandwidth issues imo. I wish that something like xet but open source could exist.
xet is open source, check out https://github.com/huggingface/xet-core.
We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend.
The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.