Not exactly (site:gitlab.open-xchange.com in a search engine gets you the links to access the projects directly), but the explore page is indeed restricted.
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Not exactly (site:gitlab.open-xchange.com in a search engine gets you the links to access the projects directly), but the explore page is indeed restricted.
You can thank AI bots for that.
Wow didn't realize it has comes to this. I guess the AI bots are ignoring the robots.txt now days huh, so just removing the index seems enough to stop the AI scrappers?
Yep. I have my own small gitweb server, and I was getting literally thousands requests per second from bots who try to crawl through all the links.