Comment by jeroenhd

If all bots are subject to a rate limit, then the system works as designed. Especially if site operators can block bot accounts. Requiring accounts is one of the easiest solutions for that problem. One of the large issues with scrapers is that they pretend to be normal internet visitors that never visited your site before, because any bot that stored cookies would immediately be rate limited by basic config.

Turnstile isn't something Cloudflare put up to annoy you. It's what the website owners decided to put up, for many different reasons.

In the same vein, Anubis has a default configuration that lets honest scrapers and crawlers through, because those can easily be rejected by basic web server configurations. Only scrapers pretending to be browsers need to solve the proof-of-work puzzle. You can disable that feature, of course.

Cloudflare may play this smart: force bots to pay for access, then take 30% of the cut and give the rest to the website owners. That way, websites get paid when the AI slop machine digests their content. Normal visitors get in for free, turn the scraper hellscape into a sustainable model. Bonus points for letting websites set their own rates (pre-declared to scrapers, of course) to dissuade all but the most interested scrapers.