# GPTBot: OpenAI Crawler, User Agent and robots.txt

> GPTBot user agent, robots.txt rules, IP ranges and what a block changes. Blocking GPTBot does not remove a site from ChatGPT search.

GPTBot is the OpenAI crawler that collects content that can be used to train OpenAI's generative AI foundation models. If you do not want your content used for this training, disallow GPTBot in robots.txt. This block does not remove your site from ChatGPT search, which uses a separate bot, OAI-SearchBot.

## User-agent string

OpenAI's [bots page](https://developers.openai.com/api/docs/bots) gives this example string and says that the version number can change:

```text
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
```

When GPTBot fetches robots.txt files, OpenAI can add a `robots.txt` marker to the string. The marker helps you find robots.txt requests in logs that do not show paths.

The robots.txt token is `GPTBot`. A token is the name in a robots.txt `User-agent` line.

## What GPTBot is for

OpenAI says it uses GPTBot to make its generative AI foundation models more useful and safe. GPTBot crawls content that can be used in training these models. OpenAI says that a disallow rule for GPTBot tells OpenAI not to use the site's content in that training.

## Block GPTBot in robots.txt

If you want to keep GPTBot out of your whole site, add this group to `/robots.txt`:

```text
User-agent: GPTBot
Disallow: /
```

If you want to keep it out of one folder only, disallow that path:

```text
User-agent: GPTBot
Disallow: /members/
```

OpenAI says each bot setting is independent of the others. For example, you can stay in ChatGPT search and opt out of training:

```text
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /
```

If you allow both bots, OpenAI can use the results of one crawl for both uses.

## What blocking GPTBot does not change

A GPTBot rule does not control ChatGPT search. [OAI-SearchBot](/ai-crawler-bot/oai-searchbot/) controls whether your site appears in [ChatGPT search](/ai-search-engine/chatgpt-search/) answers. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, but they can still appear as navigational links.

A GPTBot rule also does not stop [ChatGPT-User](/ai-crawler-bot/chatgpt-user/). That agent visits pages when a ChatGPT user asks for them.

The bots page does not say whether a block removes content that GPTBot collected before the block.

## Verify GPTBot requests

OpenAI publishes the GPTBot IP ranges at [openai.com/gptbot.json](https://openai.com/gptbot.json). Make sure that the IP address of a request is in this list before you treat it as GPTBot. The bots page does not describe a reverse DNS method.

If you want to block GPTBot in a firewall or CDN, you can match the `GPTBot` user agent. Other clients can copy the string. Only the published IP ranges show that a request comes from OpenAI.

## OpenAI bots compared

| User agent | Purpose | robots.txt | IP list |
|---|---|---|---|
| `GPTBot` | Content for training foundation models | Controlled by the `GPTBot` token | [gptbot.json](https://openai.com/gptbot.json) |
| `OAI-SearchBot` | Shows sites in ChatGPT search features | Controlled by the `OAI-SearchBot` token | [searchbot.json](https://openai.com/searchbot.json) |
| `ChatGPT-User` | Visits pages for user actions in ChatGPT and custom GPTs | OpenAI says rules do not always apply | [chatgpt-user.json](https://openai.com/chatgpt-user.json) |
| `OAI-AdsBot` | Reviews landing pages of ads submitted to ChatGPT | Not stated | [adsbot.json](https://openai.com/adsbot.json) |

OpenAI says ChatGPT-User is not used to decide whether content appears in search. It also says OAI-AdsBot visits only pages submitted as ads, and that its data is not used to train foundation models. For search results, OpenAI says a robots.txt change can take about 24 hours to apply.

Some logs and robots.txt templates also show the name `OAI-Research`. OpenAI does not document a bot with this name, so a robots.txt group for it has no documented effect.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/gptbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)