Comment by deaton 10 hours ago What you gain? Nothing, but they and other AI companies have decided not to respect your robots.txt 1 comment deaton Reply loloquwowndueo 10 hours ago There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.
loloquwowndueo 10 hours ago There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.
There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.