Skip to main content
 

Internet Archive moves toward ignoring Robots.txt for more commonsensical archival procedure.
http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives