Comment by Wowfunhappy

> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.

That is super interesting, thank you!

> At that price point, its actually very affordable to many thousands of organizations to get their own copy.

I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?