← Back to context

Comment by Ma8ee

3 days ago

I'd rather let Archive block access to that specific article for a while, but still archiving from the start.

3 comments

Ma8ee

Reply
ajb  3 days ago

In effect, robots.txt should have an "embargo" directive?

  • Bender  2 days ago

    I dont know if this is still the case but if I told IA via robots.txt not to archive my site, it would still crawl it, archive it but not display it until I shut the site down. Once robots.txt was no longer reachable they would display the archived content. The only way to stop that was to start the site back up making robots.txt reachable and wait for them to crawl it again.