# AI2Bot: Ai2 Crawler User Agent and robots.txt

> AI2Bot is the crawler of Ai2 (Allen Institute for AI). It collects content to train open models such as Olmo. User agent and how to block it.

AI2Bot is the web crawler of Ai2, the Allen Institute for AI. Ai2 says that the content it collects is used to train open language models, such as the Olmo models. If you do not want your content in that training data, block `AI2Bot` in robots.txt.

## What AI2Bot collects

The [Ai2 crawling notice](https://allenai.org/crawler) says that AI2Bot visits certain domains to find web content, and that Ai2 uses this content to train open language models.

The [Olmo 3 technical report](https://arxiv.org/abs/2512.13961) gives more detail about two crawls:

- A crawl of academic PDF files. The crawler started from academic sites and paper repositories. It collected 238 million unique PDF documents, with a cutoff date of December 2024.
- A crawl of science, education and general websites from September 12, 2024 to June 3, 2025. The start list came from manual lists of high-value sites.

The report says that the crawler identifies itself as AI2Bot, follows robots.txt and does not get past paywalls. The second crawl used the same rules. Ai2 used data from both crawls to train Olmo 3.

## User-agent string

Ai2 says that its crawler always sends this user agent:

```
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)
```

The robots.txt token is `AI2Bot`.

## Block AI2Bot in robots.txt

If you want to keep your site out of AI2Bot crawls, add this group:

```
User-agent: AI2Bot
Disallow: /
```

Ai2 also says that you can use its user agent string to filter or reject its traffic. If you want a block that does not depend on robots.txt, reject requests whose user agent contains `AI2Bot` in your web server or firewall.

## What blocking does not change

The Olmo 3 report says that most of its pretraining text comes from Common Crawl, not from AI2Bot. A block for AI2Bot does not stop the Common Crawl crawler, CCBot. If you want to keep future pages out of Common Crawl, read the [CCBot guide](/ai-crawler-bot/ccbot/).

## Verify AI2Bot requests

Ai2 does not publish IP addresses or a reverse DNS name for AI2Bot on its crawling notice. You can only match the user agent string, and any client can copy it. For questions about the crawler, Ai2 gives a contact link on the crawling notice.

## Related names

Third-party bot lists, such as [Known Agents](https://knownagents.com/agents/ai2bot-dolma), also show the names `Ai2Bot-Dolma` and `Ai2Bot-DeepResearchEval`. Ai2 does not document these names, and its crawling notice names only AI2Bot. A group for `AI2Bot` does not cover `Ai2Bot-Dolma`. If you want to block both names, add a group for each:

```
User-agent: AI2Bot
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /
```

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/ai2bot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)