Comment by mitxela
13 hours ago
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
13 hours ago
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.