Comment by chii

> many anti-scraping services explicitly allow access to Google and maybe couple of other search engines.

because google (and the couple of other search engines) provide enough value that offset the crawler's resource consumption.