Comment by reconnecting

I believe there is a slight misunderstanding regarding the role of 'AI crawlers'.

Bad crawlers have been there since the very beginning. Some of them looking for known vulnerabilities, some scraping content for third-party services. Most of them have spoofed UAs to pretend to be legitimate bots.

This is approximately 30–50% of traffic on any website.