archive.org_bot: Internet Archive Crawler and robots.txt

archive.org_bot is the crawler of the Internet Archive. It saves copies of public web pages for the Wayback Machine. The Internet Archive began to archive the web in 1996. It is not a search engine crawler. If you want your site kept out of the archive, a robots.txt rule can help, but the Internet Archive does not promise to follow it. To remove pages that it already saved, send a request by email.

What archive.org_bot does

The Internet Archive describes the bot on its archive.org_bot page. It says that the user agent archive.org_bot is used for its wide crawl of the web. It also says that it tries to crawl slowly enough not to disrupt normal web traffic. If the bot causes problems on your site, the same page gives a contact address: [email protected].

The Internet Archive help center says that its crawls find sites that many other sites link to. There is no form to request a crawl of your site. A page that no other page links to, or a page behind a password, usually does not get archived.

User agent and verification

The user agent string contains the token archive.org_bot. Use this token in robots.txt and when you search your access logs.

The Internet Archive does not publish an IP range list or a DNS verification method for this bot on the pages linked here. For this reason, you cannot prove that a request with this user agent came from the Internet Archive. Anyone can copy a user agent string.

For the older ia_archiver user agent, see the ia_archiver guide.

Block archive.org_bot in robots.txt

If you want to ask the bot not to crawl your site, add this group to robots.txt:

User-agent: archive.org_bot
Disallow: /

This rule is a request, not a guarantee. The help center says that some sites are not in the archive because robots.txt blocked the crawl. But in April 2017, the director of the Wayback Machine wrote about robots.txt. He said that the Internet Archive no longer used robots.txt files on U.S. government and military sites. The post also said that the Internet Archive planned to do this more broadly.

If you must keep the bot out, also block the user agent on your server or firewall. A server block stops only requests that carry the user agent.

Remove pages from the Wayback Machine

The documented way to remove pages that the archive already holds is a request by email. Send it to [email protected], as the help center describes. Include this information:

1. The URL or URLs of the material.
2. The time period that you want excluded.
3. The time period in which you controlled the site or the account, if applicable.
4. Other information that helps the team understand the request.

The Internet Archive reviews each request. It does not promise a result in advance.

AI and the Wayback Machine

Some publishers now block the Wayback Machine because they fear that AI companies can reuse archived copies of their pages. The Internet Archive gives its view in FAQ: Publishers Blocking the Wayback Machine. If your concern is AI training, block the AI crawlers by name as well, for example GPTBot and CCBot.