Comment by a_t48
1 day ago
The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)
1 day ago
The real GH LFS cost is not the storage but the bandwidth on pulling objects down for every fresh clone. $$$$$. See my other comment. :)
xet on hugging face does seem to not have such bandwidth issues imo. I wish that something like xet but open source could exist.
xet is open source, check out https://github.com/huggingface/xet-core.
We (I'm on the xet team at HF) are open sourcing our spec + protocol in the coming months. So, with the spec and protocol open-sourced, anyone can create xet clients and implement the protocol to build a xet backend.
The specific implementation of our xet backend is deeply integrated into HF backend so open-sourcing it directly wouldn't be very helpful. Once we get the spec + protocol released it should be easy to generate a compatible backend.
Hm alright, so in the meanwhile, the best place to use xet might be huggingface
I am wondering since a lot of people have said that for things like gamedev etc., git isn't best because of lfs issues and xet fixes it, could I use huggingface as a repository if I am gamedev or would that go against since huggingface is for AI stuff?
Also thanks for open sourcing it, I hope that it would be in permissive license and open sourcing the spec + protocol might actually push the git community to be better which is really nice!