Comment by mmarian

I'm not sure it'll work though. Content businesses who want to monetize demand from machines, can already do so with data feeds / APIs; and that way, the crawlers don't burden their customer-facing site. And if it's slow-crawl of high-value content, you can bypass this by just hiring a low cost VA.

Is there anything I'm missing?