Comment by wtf242
2 years ago
This problem is only going to get worse. for my thegreatestbooks.org site i used to just get indexed/scraped by google and bing. now it's like 50+ AI bots scraping my entire site just so they can train a LLM to answer questions my site answers without having a user ever visit my site. I just checked cloudflare and in the past 24 hours I've had 1.2 million bot/automated requests
There's a new setting in Cloudflare to block AI/scraper bots. https://blog.cloudflare.com/declaring-your-aindependence-blo...
Anyone have any experience with this? Is there nothing but upside in blocking these bots
Considering it's Buttflare, enabling it probably also means blocking random users. But of course that's not Buttflare's problem because it's not enabled by default.