Comment by bombcar
11 hours ago
This can be done by Archive.org doing it for whatever reason (asked, on their own, etc) or it can be triggered by the current owner of the domain modifying robots.txt I believe.
11 hours ago
This can be done by Archive.org doing it for whatever reason (asked, on their own, etc) or it can be triggered by the current owner of the domain modifying robots.txt I believe.
Yes: https://gist.github.com/danielhickman/38d6c7599d71c3e7dc48ca...