Comment by edg5000

Residential proxies are the only way to crawl and scrape. It's ironic for this article to come from the biggest scraping company that ever existed!

If you crawl at 1Hz per crawled IP, no reasonable server would suffer from this. It's the few bad apples (impatient people who don't rate limit) who ruin the internet for both users and hosters alike. And then there's Google.