Comment by tonymet
7 months ago
Scraping isn’t an industry. There are legitimate and illegitimate scraping pursuits.
There are lots of healthy / productive businesses in the cloud and lots of scumbags, just like any enterprise.
I still have no idea about your point, by the way.
If you have a "legitimate scraping pursuit", identify yourself appropriately that way. I'm happy to let most well-behaved scrapers access my content.
Hiding behind a residential proxy and using random user agents? Gross. Learn what consent is.
try scraping any of the major players e.g. Amazon without residential proxy it won't work. I appreciate that you are offering to abide by crawling etiquette (e.g. robots.txt) but no major app supports that any more.
You're thinking about the case of big AI companies crawling your blog. I'm talking about a small startup trying to do traditional indexing and needing to run from residential proxy to make it work.