Comment by input_sh
6 hours ago
Aren't you choosing to ignore something very specific specified in that article? Why do you make it seem that article implies it's their overall policy?
> A few months ago we stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org).
> Aren't you choosing to ignore something very specific specified in that article?
Of course not, did you ignore the lines right after? “As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.”
The announcement is from 9 years ago. I already mentioned they ignored the robots.txt for my own blog.