Comment by saaaaaam

Right, yes, I see your point. I was thinking more from the point of view of "using AI to explore and then write custom scrapers where relevant" rather than just blanket scraping. But you're right - at the scale we're talking, it's presumably just blunt-force "point-and-go" scraping, rather than anything more nuanced.

The point you make about having some sort of indicator that scrapers can follow to scrape in an optimal way (or access a dump) makes a lot of sense for people who want their content to be ingested by AI.