Comment by alphan0n

19 days ago

Can someone point out the authors robots.txt where the offense is taking place?

I’m just seeing: https://pod.geraspora.de/robots.txt

Which allows all user agents.

*The discourse server does not disallow the offending bots mentioned in their post:

https://discourse.diasporafoundation.org/robots.txt

Nor does the wiki:

https://wiki.diasporafoundation.org/robots.txt

No robots.txt at all on the homepage:

https://diasporafoundation.org/robots.txt

13 comments

alphan0n

Reply
denschub  19 days ago

the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt.