AI2Bot: Ai2 Crawler User Agent and robots.txt
AI2Bot is the crawler of Ai2 (Allen Institute for AI). It collects content to train open models such as Olmo. User agent and how to block it.
AI2Bot is the web crawler of Ai2, the Allen Institute for AI. Ai2 says that the content it collects is used to train open language models, such as the Olmo models. If you do not want your content in that training data, block AI2Bot in robots.txt.
What AI2Bot collects
The Ai2 crawling notice says that AI2Bot visits certain domains to find web content, and that Ai2 uses this content to train open language models.
The Olmo 3 technical report gives more detail about two crawls:
- A crawl of academic PDF files. The crawler started from academic sites and paper repositories. It collected 238 million unique PDF documents, with a cutoff date of December 2024.
- A crawl of science, education and general websites from September 12, 2024 to June 3, 2025. The start list came from manual lists of high-value sites.
The report says that the crawler identifies itself as AI2Bot, follows robots.txt and does not get past paywalls. The second crawl used the same rules. Ai2 used data from both crawls to train Olmo 3.
User-agent string
Ai2 says that its crawler always sends this user agent:
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)The robots.txt token is AI2Bot.
Block AI2Bot in robots.txt
If you want to keep your site out of AI2Bot crawls, add this group:
User-agent: AI2Bot
Disallow: /Ai2 also says that you can use its user agent string to filter or reject its traffic. If you want a block that does not depend on robots.txt, reject requests whose user agent contains AI2Bot in your web server or firewall.
What blocking does not change
The Olmo 3 report says that most of its pretraining text comes from Common Crawl, not from AI2Bot. A block for AI2Bot does not stop the Common Crawl crawler, CCBot. If you want to keep future pages out of Common Crawl, read the CCBot guide.
Verify AI2Bot requests
Ai2 does not publish IP addresses or a reverse DNS name for AI2Bot on its crawling notice. You can only match the user agent string, and any client can copy it. For questions about the crawler, Ai2 gives a contact link on the crawling notice.
Related names
Third-party bot lists, such as Known Agents, also show the names Ai2Bot-Dolma and Ai2Bot-DeepResearchEval. Ai2 does not document these names, and its crawling notice names only AI2Bot. A group for AI2Bot does not cover Ai2Bot-Dolma. If you want to block both names, add a group for each:
User-agent: AI2Bot
Disallow: /
User-agent: Ai2Bot-Dolma
Disallow: /