# Claude Web Search and Web Fetch: How Claude Cites Pages

> How Claude finds, reads and cites web pages in its apps and in the API, which Anthropic bots to allow, and how to keep search but block training.

Claude is the AI assistant from Anthropic. It can search the web, read pages that a user gives it and cite its sources. The same search and fetch features are also tools that developers add to their own apps through the Claude API. Anthropic runs three bots with separate robots.txt tokens, so you can block model training and still let Claude find and read your pages.

## Web search and web fetch in the Claude apps

Anthropic's [web search help page](https://support.claude.com/en/articles/10684626-enabling-and-using-web-search) says that Claude uses a search tool when a question needs current information. Every response that uses web search includes citations. A citation is a link from a statement in the answer to its source page. Claude gives direct citations, source links and, when relevant, quotes.

When web search is on, Claude can also read a page from a URL that the user gives. Anthropic calls this web fetch. On Team and Enterprise plans, an owner must turn on web search for the organization first.

Image results come from Bing, and each image has a link to its source page.

## Web search in the Claude API

Developers can give Claude the [web search tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-search-tool) in their own apps. Claude decides when to search, the API runs the search, and Claude answers with cited sources. A search can repeat several times in one request.

These parts of the API response refer to your page:

- Each search result has the `url` and `title` of the page, and a `page_age` value that tells when the site was last updated.
- Citations are always on for web search. Each citation has the `url`, the `title` and a `cited_text` field with up to 150 characters of the page text.
- A developer can limit results with `allowed_domains` or `blocked_domains`, but not both.

Anthropic tells developers to include citations to the original source when they show API output directly to end users.

## Web fetch in the Claude API

The [web fetch tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool) reads the full content of one web page or PDF. The API documentation gives these limits:

- Claude can only fetch a URL that appeared earlier in the conversation, for example in a user message or in a previous search result.
- The tool does not support websites that render their content with JavaScript.
- It reads text, HTML and PDF files only.
- A fetch can be refused because of robots.txt.
- The tool caches results, so Claude can get an older version of your page.

Citations are optional for web fetch and off by default. If your page builds its main text with JavaScript, the fetch tool cannot read that text. Serve the main text in the HTML.

## Which bots to allow or block

Anthropic's [crawler help page](https://support.claude.com/en/articles/8896518) lists three bots. Each token controls only its own bot:

- `ClaudeBot` collects content that Anthropic can use for model training. See the [ClaudeBot guide](/ai-crawler-bot/claudebot/).
- `Claude-SearchBot` indexes pages for search results in Claude. See the [Claude-SearchBot guide](/ai-crawler-bot/claude-searchbot/).
- `Claude-User` opens a page when a person asks Claude a question. See the [Claude-User guide](/ai-crawler-bot/claude-user/).

A common choice is to keep your pages in Claude's search results and out of training data:

```
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /
```

The bot guides give the details for each token, such as how to verify requests and how to contact Anthropic.

---

[View the canonical page](https://aichatwatch.com/ai-search-engine/claude/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)