Comment by baby_souffle
8 hours ago
> Who uses service providers like this?
I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS. I always set up price monitoring for any big ticket item I'm considering like appliances so I can see how their pricing changes over time. I also use scrapers for websites that don't have an API. I like having all of my purchase history indexed in a database where I can do analysis.
> These kinds of services inevitably make the web more human-hostile and expensive.
I would rather not have to spend more time circumventing stupid bot detection things. I would be more than happy to pay for access to some of this data that I cannot access any other way.. but sure, let's keep burning resources on a cat and mouse game that scrapers will always be able to win.
> I use change detection to monitor all sorts of websites for changes. Some of my favorite authors don't have RSS.
Have you considered offering, as penitence, a public feed to share the information that this process produces?
The litmus test here is whether they support https://blog.cloudflare.com/introducing-pay-per-crawl/ out of the box or not
They do not.
I don't know who 'they' is here but that's not the point? I would bet that a decent chunk of scraping happens because there is no API (or other machine-focused interface, like RSS).
"pay to crawl" sounds like the absolutely laziest possible way that a particular site could bolt on an API.
has anyone been using this for success? wondering what kinds of pays they are getting. or are the crawlers just avoiding those sites.
This attitude, and by proxy this business are the epitome of selfish entitlement.
You state that you believe you deserve access to others’ resources, at their cost, despite their clear attempts to stop you from using them, simply because you want it.
Personally I consider it fair game in "price wars".
Dynamic pricing designed to extract every penny out. Then why shouldn't I be allowed to monitor your pricing changes?
I'd counter that your attitude is a techno-authoritarian one. Why should anyone have any say over how I access and use a publicly available resource? At least so long as my actions don't directly cause technical problems for the service operator.
You can already access these resources. What does it matter if you do the clicking or you have headless chrome do the clicking while you make a cup of coffee?
Dude, it's a web request. It's not that deep.