← Back to context Comment by cyberax 1 day ago You can thank AI bots for that. 2 comments cyberax Reply zenmac 19 hours ago Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers? cyberax 14 hours ago Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.
zenmac 19 hours ago Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers? cyberax 14 hours ago Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.
cyberax 14 hours ago Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.