Comment by JohnFen
6 days ago
No, sitemap serves a similar but different purpose. The design goal of robots.txt is to guide crawlers to the parts of the site that are worth crawling. That it's used as a very weak method of access control is a hack.
No comments yet
Contribute on Hacker News ↗