Comment by jeroenhd

5 hours ago

If all bots are subject to a rate limit, then the system works as designed. Especially if site operators can block bot accounts. Requiring accounts is one of the easiest solutions for that problem. One of the large issues with scrapers is that they pretend to be normal internet visitors that never visited your site before, because any bot that stored cookies would immediately be rate limited by basic config.

Turnstile isn't something Cloudflare put up to annoy you. It's what the website owners decided to put up, for many different reasons.

In the same vein, Anubis has a default configuration that lets honest scrapers and crawlers through, because those can easily be rejected by basic web server configurations. Only scrapers pretending to be browsers need to solve the proof-of-work puzzle. You can disable that feature, of course.

Cloudflare may play this smart: force bots to pay for access, then take 30% of the cut and give the rest to the website owners. That way, websites get paid when the AI slop machine digests their content. Normal visitors get in for free, turn the scraper hellscape into a sustainable model. Bonus points for letting websites set their own rates (pre-declared to scrapers, of course) to dissuade all but the most interested scrapers.

3 comments

jeroenhd

Reply
Hackbraten  3 hours ago

> Normal visitors get in for free

Except for the unfortunate minority of normal visitors who always get misclassified as bots and get denied access regularly.

I wouldn’t be complaining if Cloudflare’s misclassifier bit any user with the same small probability. But it keeps biting the same users over and over again.

  • jeroenhd  2 hours ago

    It'll bite any user that the bots will copy. Every trick a user might use to make themselves unrecognisable on the web is also used by a bot farm somewhere out there.

    Website owners that use Turnstile and other such services choose to exclude these users. A tiny margin of false positives isn't going to dissuade most website owners, I imagine only the ones that themselves have issues with Cloudflare will bother to add the necessary rules to permit uncommon users (and the bots that copy them).