Comment by mmis1000

The whole reason of anubis is the bot don't make a damn shit about whether whole data is accessible or not, and even crawl dynamic links in robots.txt in high frequency.

Even wikipedia begged for those damn bot about stopping doing this, the data is already accessible in archive here.