# Claude-SearchBot: Anthropic Search Crawler and robots.txt

> Claude-SearchBot indexes pages for search results in Claude. Its robots.txt token, what a block changes and how to verify requests.

Claude-SearchBot is an Anthropic bot that crawls the web to improve the search results that Claude gives its users. If you want Claude's web search to find your pages, allow `Claude-SearchBot` in robots.txt. If you block it, your site can become less visible in those results.

## What Claude-SearchBot does

Anthropic's [crawler help page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) says that Claude-SearchBot analyzes online content to make search responses more relevant and accurate. Cloudflare puts it in the "AI Search" category of its [AI Crawl Control bot list](https://developers.cloudflare.com/ai-crawl-control/reference/bots/).

This bot has one job: search indexing. Model training belongs to [ClaudeBot](/ai-crawler-bot/claudebot/), and page visits for single user questions belong to [Claude-User](/ai-crawler-bot/claude-user/).

## robots.txt token and user agent

The robots.txt token is `Claude-SearchBot`. robots.txt is a text file at the site root with rules for crawlers. No full user-agent string is published for this bot. To find it in your logs, search for `Claude-SearchBot` in the User-Agent field.

The help page says that the bots obey robots.txt and support the non-standard `Crawl-delay` rule.

## Allow Claude-SearchBot

If your robots.txt has a group for all crawlers (`User-agent: *`) with `Disallow: /`, that group also blocks Claude-SearchBot. Under the robots.txt standard, a crawler follows the group that names it, and it uses the `*` group only when no group names it. To let this bot index your site, give it its own group:

```
User-agent: Claude-SearchBot
Allow: /
```

A group for one token does not apply to the other tokens. So you can allow search indexing here and still block training. The [Claude guide](/ai-search-engine/claude/) shows that robots.txt example.

## Block or limit Claude-SearchBot

If you do not want this bot to index your site, add this group to the robots.txt file in the top-level directory:

```
User-agent: Claude-SearchBot
Disallow: /
```

To keep one folder out of the index, disallow only that path:

```
User-agent: Claude-SearchBot
Disallow: /internal/
```

If the bot sends too many requests, slow it down instead of a block:

```
User-agent: Claude-SearchBot
Crawl-delay: 2
```

Add the rule on each subdomain that you want to opt out. A rule on `example.com` does not cover `shop.example.com`.

## What a block changes

A block stops the indexing of your content for search. The help page says that this can reduce the visibility and accuracy of your site in search results for Claude users.

The block does not stop Claude from opening a page when a user asks for it, and it says nothing about training.

## Verify requests

Anthropic lists the IP ranges of its bots at [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json). If the address of a request is not in this list, treat the request as unverified, even when the user agent says Claude-SearchBot.

Do not use an IP block as your opt-out. The block also stops the bot from reading your robots.txt, so the opt-out can fail. If you think that the bot does not obey your rules, write to claudebot@anthropic.com from an address at your domain.

For Claude as a product, see the [Claude guide](/ai-search-engine/claude/).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/claude-searchbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)