April 30, 2017
Internet Archive moves toward ignoring Robots.txt for more commonsensical archival procedure. http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives
