Comment by zenmac
19 hours ago
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
19 hours ago
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.