Comment by cookiengineer
2 days ago
Anna's archive has the metadata on it.
IA was around 300TB last time I checked.
libgen was around 190TB. For my own at home cluster I decided to go for 512TB but I can't host nor upload in these bandwidth requirements from here.
I started to build sth like a torrent splitter tool yesterday because I realized that all torrent clients just crash when you try to open, modify, or seed those torrents.
Edit: correction, the IA is ~15PB big, brewster kahle mentioned it in the documentary (2014)
300tb seems insanely small so that 15pb number is probably closer
Id say though with the rise of video on social media, if they didn't 3x or 4x that 15pb number since then.
I'm sure there's plenty of room to replicate to aws but I'm sure they couldn't afford to host 60pb of data on aws. They have their own Colo or data center if I recall.