Comment by Reason077 4 hours ago Do we need a "robots must respect robots.txt" law? 1 comment Reason077 Reply WhyNotHugo 3 hours ago If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.(Disallowing * isn’t usually an option since it makes you disappear from search engines).
WhyNotHugo 3 hours ago If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.(Disallowing * isn’t usually an option since it makes you disappear from search engines).
If we did, bot authors would comply by just changing their User-Agent to something different that’s not expressly forbidden.
(Disallowing * isn’t usually an option since it makes you disappear from search engines).