Comment by denschub
19 days ago
the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt.
https://web.archive.org/web/20240101000000*/https://wiki.dia...
notice how there's a period of almost two months with no new index, just until a week before I posted this? I wonder what might have caused this!!1
(and it's not like they only check robots.txt once a month or so. https://stuff.overengineer.dev/stash/2024-12-30-dfwiki-opena...)
:/ Common Crawl archives robots.txt and indicates that the file at wiki.diasporafoundation.org was unchanged in November and December from what it is now. Unchanged from September, in fact.
https://pastebin.com/VSHMTThJ
https://index.commoncrawl.org/
9 replies →