Comment by usef-

Their scraping products are mostly about paying them money to not be blocked.

I mean that they're telling developers that they should use Cloudflare's platform to build agents, the kind of agents that would go across the web and act on behalf of users... but then they're also the ones blocking those requests.

This always engenders a solid amount of distaste from me, because much like Google and Chrome, it creates the incentive for you to treat yourself better than others. Especially coupled with the trust stuff. Of course, Cloudflare is always going to trust their own platform.

> not take down systems

"Block on pages with ads" is probably about preventing the AI crawlers from clicking on the ads which maybe considered cheating by the ad company.

If you want to prevent "bot attacks", maybe the "Block" option will do the trick.

But of course, to do all that you need to put some trust on Cloudflare, because they're the one identifying the bots from normal users.

For me, as someone who's hosting a Gitea instance behind Cloudflare, I have a Configuration Rule set that says: if the client is trying to access a URL that is beyond certain length limit, then trigger "Browser Integrity Check" and "I’m Under Attack", a.k.a stricter security checks.

The match expression of the rule looked something like this:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/"
    )

(The `!!!!!SET LENGTH LIMIT!!!!!` is an integer of the length limit you wanted to set)

This rule alone basically blocked all abusive bot traffic to almost zero for my site (https://i.imgur.com/LaOjjvV.png, see the traffic drop around 10 clock and Cloudflare mitigation kicks in).

But of course, you need to figure out your own rules based on the characteristic of the website. Also, you can be more creative: for example, my actual rule is more complex than that, it also checks to see if a cookie is not set, and only triggers when all condition are met:

    (
      len(http.request.uri) > !!!!!SET LENGTH LIMIT!!!!! and
      not lower(http.request.uri.path) contains ".git/" and
      not lower(http.request.uri.path) contains "api/" and
      not http.cookie wildcard "*!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!*"
    )

then, as part two of that rule, I have a Response Header Transform Rules that says:

    (
      len(http.request.uri) <= !!!!!SET LENGTH LIMIT!!!!! and
      not http.cookie contains "!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!"
    )

and if this Response Header Transform Rules is triggered, it sets the cookie `!!!!!COOKIE NAME!!!!!=!!!!!COOKIE VALUE!!!!!`.

(Note: `!!!!!COOKIE NAME!!!!!` and `!!!!!COOKIE VALUE!!!!!` are the variables you need to customize)

When you put the two rules together, it forces clients to access "shallow" (short URL) pages first as an user would normally do, before they can access "deeper" (long URL) content hosted on the site without triggering more strict security checks. If that makes sense.

Also, don't forget the cookie basically also dug a hole in the security setting. So it's really a balance between avoid annoying the user and protect your site. You need to be smart and be flexible about it, otherwise your users will just leave.