Comment by Wowfunhappy
5 hours ago
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
5 hours ago
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
There are many more labs than that, and humans aren't designing unique scraping processes per domain.