Comment by mitxela

4 hours ago

robots.txt is a shitshow just like user agents. It's been twisted so many ways it doesn't reliably signal actual intent any more.

0 comments