Comment by arjie
14 hours ago
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
That is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
One of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
That makes sense, Xe, but you don't enable it at:
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.