ClaudeBot: Anthropic Crawler, User Agent and robots.txt

ClaudeBot is the web crawler that Anthropic uses to collect public web content for the training of its Claude models. If you do not want your future content in that training data, disallow ClaudeBot in robots.txt. Anthropic runs two more bots with their own tokens, so decide on each one separately.

What ClaudeBot does

Anthropic’s crawler help page says that ClaudeBot collects web content that can become training data for its generative AI models. When a site blocks ClaudeBot, Anthropic treats the block as a signal to exclude the site’s future content from its training datasets.

Anthropic says that its bots obey robots.txt (a text file at the site root with rules for crawlers). The bots also do not try to get past CAPTCHAs. Cloudflare puts ClaudeBot in its “AI Crawler” category in the AI Crawl Control bot list.

Other Anthropic bots

Anthropic runs two more bots with their own tokens. Claude-SearchBot indexes pages for search results in Claude, and Claude-User opens a page when a user asks for it. A ClaudeBot rule does not change what these two bots do.

Retired tokens: anthropic-ai and Claude-Web

anthropic-ai and Claude-Web are two older Anthropic user agent names. In July 2024, 404 Media reported that many sites blocked these two names in robots.txt but did not block ClaudeBot. After the report, an Anthropic spokesperson said that the two names are no longer in use. The spokesperson also said that ClaudeBot obeys robots.txt rules that sites wrote for the old names.

Anthropic’s current help page does not mention either name, and it does not repeat that statement. If your robots.txt has a group for anthropic-ai or Claude-Web, keep it, but also add groups for the current tokens. If either old name shows up in your logs, make sure that the IP address is in the Anthropic IP list below.

User agent

The user agent is the client name in the User-Agent HTTP header. Anthropic’s help page gives the robots.txt token ClaudeBot but does not publish a full user-agent string. To find ClaudeBot in your logs, search for the token ClaudeBot in the User-Agent field.

Block ClaudeBot in robots.txt

If you want to block ClaudeBot from the whole site, add this group to the robots.txt file in the top-level directory:

User-agent: ClaudeBot
Disallow: /

Anthropic asks you to add the rule on each subdomain that you want to opt out. A rule on example.com does not cover blog.example.com.

Slow ClaudeBot down

If ClaudeBot sends too many requests, you can use Crawl-delay instead of a block. Anthropic supports this non-standard rule and gives this example:

User-agent: ClaudeBot
Crawl-delay: 1

Verify requests

Anthropic publishes the IP ranges of its bots at claude.com/crawling/bots.json. If a request comes from an address in this list, it comes from Anthropic. If a request has ClaudeBot in the user agent but comes from another address, treat it as unverified. Anyone can copy a user-agent string.

Server and firewall blocks

Anthropic says that robots.txt is the way to opt out. An IP block can fail as an opt-out, because the block also stops the bots from reading your robots.txt. If you block Anthropic addresses on the server or firewall, keep the robots.txt rule as well.

Contact Anthropic

If you think that an Anthropic bot does not obey your rules, write to [email protected]. Anthropic asks you to send the email from an address at the domain that the report is about.