Comment by arianvanp
7 hours ago
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not
They do not.
7 hours ago
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not
They do not.
I don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS).
"pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.
has anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.