# OAI-SearchBot: OpenAI Search Crawler and robots.txt

> OAI-SearchBot finds pages for ChatGPT search. User-agent string, robots.txt rules, how it differs from GPTBot, and OpenAI IP ranges.

OAI-SearchBot is the OpenAI crawler that finds web pages for the search features of ChatGPT. If you want your pages in ChatGPT search answers, allow `OAI-SearchBot` in robots.txt and let requests from OpenAI's published IP ranges through. If you block it, ChatGPT search answers do not show your pages.

## What OAI-SearchBot does

OpenAI's [crawler page](https://developers.openai.com/api/docs/bots) says that OAI-SearchBot is used to show websites in search results in ChatGPT's search features. OpenAI says that sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. These sites can still appear as navigational links.

The [ChatGPT Search guide](/ai-search-engine/chatgpt-search/) explains the product that this crawler supports.

## User-agent string

OpenAI gives this example user-agent string. OpenAI says that the version number can change:

```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
```

When OAI-SearchBot fetches robots.txt, it can add a `robots.txt` marker to the string:

```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot
```

The marker helps you tell robots.txt requests apart from other requests when your logs do not show paths. The robots.txt token is `OAI-SearchBot`.

## Allow or block OAI-SearchBot in robots.txt

OpenAI recommends that you allow OAI-SearchBot if you want your site in search results:

```
User-agent: OAI-SearchBot
Allow: /
```

To keep your site out of ChatGPT search answers, use this group:

```
User-agent: OAI-SearchBot
Disallow: /
```

To keep only one folder out of search, disallow that path:

```
User-agent: OAI-SearchBot
Disallow: /drafts/
```

OpenAI says that its search systems take about 24 hours to adjust after you change robots.txt.

## Other OpenAI user agents

OpenAI says that each robots.txt setting is independent of the others. A rule for OAI-SearchBot does not change what [GPTBot](/ai-crawler-bot/gptbot/), the training crawler, or [ChatGPT-User](/ai-crawler-bot/chatgpt-user/), the agent for user actions, can do. If you allow both OAI-SearchBot and GPTBot, OpenAI can use the results of one crawl for both purposes.

## Identify real requests

Anyone can copy a user-agent string. OpenAI publishes the IP ranges of OAI-SearchBot in [searchbot.json](https://openai.com/searchbot.json). Make sure that the IP address of a request is inside one of the listed prefixes. OpenAI's crawler page does not describe a reverse DNS method.

The file has a `creationTime` field and a `prefixes` list with one `ipv4Prefix` value for each range. If `jq` is on your computer, this command prints the ranges:

```
curl -s https://openai.com/searchbot.json | jq -r '.prefixes[].ipv4Prefix'
```

If a firewall or CDN blocks unknown bots on your site, add these ranges to its allowlist. OpenAI recommends that you allow requests from its published IP ranges. The list changes from time to time, so update your allowlist when `creationTime` changes.

## Block OAI-SearchBot on the server

robots.txt is a request. A server rule is a firm block. This nginx example returns HTTP 403 for the token:

```nginx
if ($http_user_agent ~* "OAI-SearchBot") {
    return 403;
}
```

Put the example in the `server` block of your site configuration, then reload nginx. The rule stops only requests that carry the token. To block by address, use the ranges in searchbot.json.

## What a block changes

After a block, ChatGPT search answers do not show your pages, except as navigational links. A block does not stop ChatGPT-User, so ChatGPT can still open a page when a user asks for it. A block also says nothing about training. Only a GPTBot rule does that.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/oai-searchbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)