# ia_archiver: Alexa Crawler User Agent and robots.txt

> ia_archiver is the crawler user agent of Alexa Internet, which closed in May 2022. Its history, what the old robots.txt group does today and how to handle it.

ia_archiver is the user agent of the crawler that Alexa Internet ran. Amazon closed Alexa.com on May 1, 2022. A robots.txt rule for ia_archiver does not name the current Internet Archive crawler, archive.org_bot. If an old ia_archiver group is in your robots.txt, it does no harm, but it no longer controls an active crawler that anyone documents.

## Where the name comes from

The name started at the Internet Archive. In September 1996, the Internet Archive [wrote on the robots mailing list](https://webdoc.gwdg.de/ebook/aw/1999/webcrawler/mailing-lists/robots/1258.html) that its robot "will identify itself as "ia_archiver"". The same message said it "will obey the Standard for Robot Exclusion".

Brewster Kahle and Bruce Gilliat founded Alexa Internet in April 1996. Amazon bought it in 1999 for about 250 million US dollars in Amazon stock ([Wikipedia](https://en.wikipedia.org/wiki/Alexa_Internet)). Alexa crawled the web and gave its crawls to the Internet Archive. A 2013 Internet Archive FAQ said: "Much of our archived web data comes from our own crawls or from Alexa Internet's crawls" ([archived FAQ](https://web.archive.org/web/20130716152622/http://www.archive.org/about/faqs.php)).

In 2017, Mark Graham of the Internet Archive answered a reader on the Internet Archive blog: "The 'ia_archiver' User Agent is used by Alexa Internet, not the Internet Archive" ([blog comments](https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/)).

## What Alexa did with its data

Alexa published a traffic rank, an estimate of the popularity of a website. Per Wikipedia, Alexa based the estimate on a sample of millions of Internet users. The Global Rank combined the daily visitors and page views of a site over three months. Sites that installed an Alexa script also sent data.

The crawl had other uses too:

- In 1998, Alexa gave a copy of its archive, two terabytes in size, to the Library of Congress.
- In December 2005, Alexa opened its search index and crawl systems to third-party programs through the Alexa Web Information Service (AWIS).
- Alexa Web Search stopped new sign-ups on November 27, 2008, and closed for existing customers on January 26, 2009.

## User agent string

Public bot lists record this string ([crawler-user-agents](https://github.com/monperrus/crawler-user-agents), [Known Agents](https://knownagents.com/agents/ia-archiver)):

```
ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)
```

The robots.txt token is `ia_archiver`. The string points to an Alexa help page and an Alexa email address. Both belong to a service that closed.

## Status after Alexa closed

On December 8, 2021, Amazon [announced](https://web.archive.org/web/20220102200605/https://support.alexa.com/hc/en-us/articles/4410503838999) the end of Alexa.com on May 1, 2022. We found no statement about the crawler after that date. A request with the ia_archiver name in your logs today does not prove that Alexa or the Internet Archive sent it. Any client can send this name. We found no IP list or DNS method to verify it.

If such requests cause load on your server, block the name there. This nginx example returns HTTP 403:

```nginx
if ($http_user_agent ~* "ia_archiver") {
    return 403;
}
```

Put the example in the `server` block of your site configuration, then reload nginx.

## The old robots.txt group

Many robots.txt files still contain this group:

```
User-agent: ia_archiver
Disallow: /
```

This group names only ia_archiver. You can keep it or delete it. The Internet Archive crawls with archive.org_bot ([Cloudflare Radar](https://radar.cloudflare.com/bots/directory/archive-org-bot)), and a group for ia_archiver does not apply to that bot. The [archive.org_bot guide](/ai-crawler-bot/archive-org-bot/) covers the current crawler. It also explains how far the Internet Archive follows robots.txt and how to ask it to remove pages from the Wayback Machine.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/ia-archiver/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)