Comment by rgrieselhuber
15 hours ago
It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.
15 hours ago
It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.
I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.