cohere-ai and cohere-training-data-crawler User Agents

cohere-ai is a user-agent token that bot directories and robots.txt block lists link to Cohere, a company that makes large language models for businesses. Cohere does not document a bot with this name. Cohere says that it does not use bots to crawl the web for model training at this time. If you want to block the token anyway, add a robots.txt rule and, for a firm block, a server rule.

What Cohere documents

Cohere’s web crawlers page has a table for its crawler bots. The table lists no bots. The page says that Cohere does not use bots to crawl or scrape web content for the training of its foundation models “at this time”. If Cohere starts to do this, it will add the bots to the table with instructions to block them.

The page also says that Cohere’s policy requires its crawlers to obey robots.txt. Its example uses a placeholder token, Coherebot, not cohere-ai. Like other operators, Cohere warns that an IP block can fail as an opt-out, because it stops the crawler from reading robots.txt.

The statement covers crawls for model training. The page does not say whether Cohere products fetch pages for other purposes, for example to answer a user’s question.

What third-party lists say

Known Agents calls cohere-ai an “undocumented AI agent” that Cohere’s chat products possibly send in response to user prompts. It lists this user-agent string:

Mozilla/5.0 (compatible; cohere-ai; +https://cohere.com/crawler)

The URL in that string, https://cohere.com/crawler, returned HTTP 404 in October 2026. Other directories show different full strings. No directory gives an IP list or a reverse DNS method for cohere-ai.

cohere-training-data-crawler

A second token, cohere-training-data-crawler, also appears in bot directories. Known Agents lists it as an AI data scraper that Cohere operates. It gives the user agent as the plain string cohere-training-data-crawler. It links to no Cohere document and names no verification method. Cohere’s web crawlers page does not list this token either. Any client can send it, so a request with this name does not prove that Cohere sent it.

Block cohere-ai in robots.txt

robots.txt is a text file at the site root with rules for crawlers. If you want to block both Cohere tokens that directories report, add this group:

User-agent: cohere-ai
User-agent: cohere-training-data-crawler
Disallow: /

Cohere says to put the file in the top-level directory of each subdomain that you want to exclude. Because Cohere does not document these tokens, nobody can confirm which client reads this group. The rule does no harm if no client uses the tokens.

Block it on the server

RFC 9309 says that robots.txt rules are not a form of access authorization. If you want a firm block, return HTTP 403 for the token on your web server. This nginx example goes in the server block of your site configuration:

if ($http_user_agent ~* "cohere-ai") {
    return 403;
}

Related guides

Cohere’s chat models are covered in the Cohere Command guide.