Comment by gnfargbl

> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).

That's a hard one for Cloudflare, no? They got to where they are by being (if you want to be cynical, playing the role of) the benevolent, neutral guardians of the internet, a one-stop shop that makes most of the bad nonsense go away without much effort on the part of the developer. Continuing that stance probably does mean some basic AI crawler blocking by default, unfortunately. At least they document it [1].

[1] https://developers.cloudflare.com/bots/additional-configurat...