Comment by sva_
14 hours ago
I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.
14 hours ago
I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.
My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time.
https://developers.cloudflare.com/changelog/post/2026-07-01-...
From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it.
I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index.
I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others.
They're also starting a registry of "Approved" crawlers.