Comment by typpilol
2 days ago
Do we know how much data the internet archive has?
Is it even viable to replicate to multiple regions if it's 1000s of PB?
2 days ago
Do we know how much data the internet archive has?
Is it even viable to replicate to multiple regions if it's 1000s of PB?
Anna's archive has the metadata on it.
IA was around 300TB last time I checked.
libgen was around 190TB. For my own at home cluster I decided to go for 512TB but I can't host nor upload in these bandwidth requirements from here.
I started to build sth like a torrent splitter tool yesterday because I realized that all torrent clients just crash when you try to open, modify, or seed those torrents.
Edit: correction, the IA is ~15PB big, brewster kahle mentioned it in the documentary (2014)