Comment by deadalus
5 years ago
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
5 years ago
archive.org obeys the robots.txt exclusion but archive.today doesn't. This means that many websites(like 4chan) cannot be archived with archive.org.
That's true, so does webcitation.org, which doesn't obey robots.txt.
4chan has archives dedicated for that site anyway (Warosu, etc.).