Claude-SearchBot: Anthropic Search Crawler and robots.txt

Claude-SearchBot is an Anthropic bot that crawls the web to improve the search results that Claude gives its users. If you want Claude’s web search to find your pages, allow Claude-SearchBot in robots.txt. If you block it, your site can become less visible in those results.

What Claude-SearchBot does

Anthropic’s crawler help page says that Claude-SearchBot analyzes online content to make search responses more relevant and accurate. Cloudflare puts it in the “AI Search” category of its AI Crawl Control bot list.

This bot has one job: search indexing. Model training belongs to ClaudeBot, and page visits for single user questions belong to Claude-User.

robots.txt token and user agent

The robots.txt token is Claude-SearchBot. robots.txt is a text file at the site root with rules for crawlers. No full user-agent string is published for this bot. To find it in your logs, search for Claude-SearchBot in the User-Agent field.

The help page says that the bots obey robots.txt and support the non-standard Crawl-delay rule.

Allow Claude-SearchBot

If your robots.txt has a group for all crawlers (User-agent: *) with Disallow: /, that group also blocks Claude-SearchBot. Under the robots.txt standard, a crawler follows the group that names it, and it uses the * group only when no group names it. To let this bot index your site, give it its own group:

User-agent: Claude-SearchBot
Allow: /

A group for one token does not apply to the other tokens. So you can allow search indexing here and still block training. The Claude guide shows that robots.txt example.

Block or limit Claude-SearchBot

If you do not want this bot to index your site, add this group to the robots.txt file in the top-level directory:

User-agent: Claude-SearchBot
Disallow: /

To keep one folder out of the index, disallow only that path:

User-agent: Claude-SearchBot
Disallow: /internal/

If the bot sends too many requests, slow it down instead of a block:

User-agent: Claude-SearchBot
Crawl-delay: 2

Add the rule on each subdomain that you want to opt out. A rule on example.com does not cover shop.example.com.

What a block changes

A block stops the indexing of your content for search. The help page says that this can reduce the visibility and accuracy of your site in search results for Claude users.

The block does not stop Claude from opening a page when a user asks for it, and it says nothing about training.

Verify requests

Anthropic lists the IP ranges of its bots at claude.com/crawling/bots.json. If the address of a request is not in this list, treat the request as unverified, even when the user agent says Claude-SearchBot.

Do not use an IP block as your opt-out. The block also stops the bot from reading your robots.txt, so the opt-out can fail. If you think that the bot does not obey your rules, write to [email protected] from an address at your domain.

For Claude as a product, see the Claude guide.