Comment by danso

3 years ago

A few years ago, they realized it would be in the public archival interest to ignore robots.txt for gov/mil sites, and then possibly more types of sites. They have long had a manual process for submitting takedown requests by email; here's someone on reddit who claimed to have successfully gotten a page (presumably not under their control) removed from the archive [1]