# AI Crawler Bots: Training, Search and User Fetchers

> What AI crawler bots are and their three types: training crawlers, AI search crawlers and user-triggered fetchers, with examples and sources.

An AI crawler bot is a program that fetches web pages for an AI company. The large AI companies now run separate bots for model training, for AI search and for fetches that a user starts. If you want to control AI access, decide per type, because a rule for one bot does not change the others.

## What an AI crawler is

A crawler is a program that requests web pages with no person at a browser. Each request carries a user agent, the client name in the `User-Agent` HTTP header. Each operator also documents a robots.txt token, the name that you use in your `/robots.txt` file.

The user agent and the token are often the same word, but not always. For example, Google-Extended is a robots.txt token only. Google says it has no separate HTTP user agent string ([Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)).

## The three main types

| Type | What it does | Examples |
|---|---|---|
| Training crawler | Collects pages that can help train AI models | [GPTBot](/ai-crawler-bot/gptbot/), [ClaudeBot](/ai-crawler-bot/claudebot/) |
| AI search crawler | Indexes pages so that an AI search product can show and link them | [OAI-SearchBot](/ai-crawler-bot/oai-searchbot/), [Claude-SearchBot](/ai-crawler-bot/claude-searchbot/), [PerplexityBot](/ai-crawler-bot/perplexitybot/) |
| User-triggered fetcher | Fetches a page when a user asks the AI about it | [ChatGPT-User](/ai-crawler-bot/chatgpt-user/), [Claude-User](/ai-crawler-bot/claude-user/), [Perplexity-User](/ai-crawler-bot/perplexity-user/) |

## Training crawlers

OpenAI says GPTBot "is used to make our generative AI foundation models more useful and safe" ([OpenAI](https://developers.openai.com/api/docs/bots)). Anthropic says ClaudeBot collects web content that can contribute to the training of its models ([Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

Some tokens control AI use without a separate crawler. With Google-Extended, a site owner controls whether content that Google crawls can help train future Gemini models. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". See the [Google-Extended guide](/ai-crawler-bot/google-extended/).

## AI search crawlers

These bots build an index for an AI search product. If a site blocks them, the product can stop showing the site as a source.

- OpenAI says OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features". Sites that opt out are not shown in ChatGPT search answers, but they can still appear as navigational links.
- Perplexity says PerplexityBot is "designed to surface and link websites in search results on Perplexity". It also says the bot "is not used to crawl content for AI foundation models" ([Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)).
- Anthropic says Claude-SearchBot works to improve the quality of search results for users.

## User-triggered fetchers

A user-triggered fetcher reads a page because a person asked the AI about it. Perplexity says that Perplexity-User is not used for web crawling. Operators treat robots.txt differently for these bots:

- OpenAI says that robots.txt rules do not always apply to ChatGPT-User, because a user starts the action.
- Perplexity says that Perplexity-User "generally ignores robots.txt rules".
- Anthropic says that its bots, Claude-User included, honor robots.txt.

## Other bots in AI crawler lists

AI crawler lists often include bots that serve other products. Examples:

- Link preview bots, such as [LinkedInBot](/ai-crawler-bot/linkedinbot/).
- Web archive crawlers, such as [archive.org_bot](/ai-crawler-bot/archive-org-bot/).
- Open dataset crawlers, such as [CCBot](/ai-crawler-bot/ccbot/) of Common Crawl.
- Crawlers for software tools, such as [HubSpot Crawler](/ai-crawler-bot/hubspot-crawler/).

Some names in these lists have no operator documentation at all, for example [GrokBot](/ai-search-engine/grok/).

To find which of these bots visit your site, and to write rules for them, see [how to find and block AI crawlers on your site](/ai-crawler-bot/introduction-to-ai-crawlers/).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/introduction-to-ai-crawler-bots/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)