Comment by loloquwowndueo
11 hours ago
There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.
11 hours ago
There are other ways to block robots from crawling our sites. I have a robots.txt but place no faith in it, it’s just there because it’s cheap and does stop some of the crawlers.
No comments yet
Contribute on Hacker News ↗