# ClaudeBot: Anthropic Crawler, User Agent and robots.txt

> ClaudeBot is Anthropic's crawler for AI model training. Its robots.txt token, Anthropic's other bots, the IP list and how to block it.

ClaudeBot is the web crawler that Anthropic uses to collect public web content for the training of its Claude models. If you do not want your future content in that training data, disallow `ClaudeBot` in robots.txt. Anthropic runs two more bots with their own tokens, so decide on each one separately.

## What ClaudeBot does

Anthropic's [crawler help page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says that ClaudeBot collects web content that can become training data for its generative AI models. When a site blocks ClaudeBot, Anthropic treats the block as a signal to exclude the site's future content from its training datasets.

Anthropic says that its bots obey robots.txt (a text file at the site root with rules for crawlers). The bots also do not try to get past CAPTCHAs. Cloudflare puts ClaudeBot in its "AI Crawler" category in the [AI Crawl Control bot list](https://developers.cloudflare.com/ai-crawl-control/reference/bots/).

## Other Anthropic bots

Anthropic runs two more bots with their own tokens. [Claude-SearchBot](/ai-crawler-bot/claude-searchbot/) indexes pages for search results in Claude, and [Claude-User](/ai-crawler-bot/claude-user/) opens a page when a user asks for it. A ClaudeBot rule does not change what these two bots do.

## Retired tokens: anthropic-ai and Claude-Web

`anthropic-ai` and `Claude-Web` are two older Anthropic user agent names. In July 2024, [404 Media](https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/) reported that many sites blocked these two names in robots.txt but did not block ClaudeBot. After the report, an Anthropic spokesperson said that the two names are no longer in use. The spokesperson also said that ClaudeBot obeys robots.txt rules that sites wrote for the old names.

Anthropic's current help page does not mention either name, and it does not repeat that statement. If your robots.txt has a group for `anthropic-ai` or `Claude-Web`, keep it, but also add groups for the current tokens. If either old name shows up in your logs, make sure that the IP address is in the Anthropic IP list below.

## User agent

The user agent is the client name in the User-Agent HTTP header. Anthropic's help page gives the robots.txt token `ClaudeBot` but does not publish a full user-agent string. To find ClaudeBot in your logs, search for the token `ClaudeBot` in the User-Agent field.

## Block ClaudeBot in robots.txt

If you want to block ClaudeBot from the whole site, add this group to the robots.txt file in the top-level directory:

```
User-agent: ClaudeBot
Disallow: /
```

Anthropic asks you to add the rule on each subdomain that you want to opt out. A rule on `example.com` does not cover `blog.example.com`.

## Slow ClaudeBot down

If ClaudeBot sends too many requests, you can use `Crawl-delay` instead of a block. Anthropic supports this non-standard rule and gives this example:

```
User-agent: ClaudeBot
Crawl-delay: 1
```

## Verify requests

Anthropic publishes the IP ranges of its bots at [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json). If a request comes from an address in this list, it comes from Anthropic. If a request has `ClaudeBot` in the user agent but comes from another address, treat it as unverified. Anyone can copy a user-agent string.

## Server and firewall blocks

Anthropic says that robots.txt is the way to opt out. An IP block can fail as an opt-out, because the block also stops the bots from reading your robots.txt. If you block Anthropic addresses on the server or firewall, keep the robots.txt rule as well.

## Contact Anthropic

If you think that an Anthropic bot does not obey your rules, write to claudebot@anthropic.com. Anthropic asks you to send the email from an address at the domain that the report is about.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/claudebot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)