OAI-SearchBot: OpenAI Search Crawler and robots.txt

OAI-SearchBot is the OpenAI crawler that finds web pages for the search features of ChatGPT. If you want your pages in ChatGPT search answers, allow OAI-SearchBot in robots.txt and let requests from OpenAI’s published IP ranges through. If you block it, ChatGPT search answers do not show your pages.

What OAI-SearchBot does

OpenAI’s crawler page says that OAI-SearchBot is used to show websites in search results in ChatGPT’s search features. OpenAI says that sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers. These sites can still appear as navigational links.

The ChatGPT Search guide explains the product that this crawler supports.

User-agent string

OpenAI gives this example user-agent string. OpenAI says that the version number can change:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

When OAI-SearchBot fetches robots.txt, it can add a robots.txt marker to the string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

The marker helps you tell robots.txt requests apart from other requests when your logs do not show paths. The robots.txt token is OAI-SearchBot.

Allow or block OAI-SearchBot in robots.txt

OpenAI recommends that you allow OAI-SearchBot if you want your site in search results:

User-agent: OAI-SearchBot
Allow: /

To keep your site out of ChatGPT search answers, use this group:

User-agent: OAI-SearchBot
Disallow: /

To keep only one folder out of search, disallow that path:

User-agent: OAI-SearchBot
Disallow: /drafts/

OpenAI says that its search systems take about 24 hours to adjust after you change robots.txt.

Other OpenAI user agents

OpenAI says that each robots.txt setting is independent of the others. A rule for OAI-SearchBot does not change what GPTBot, the training crawler, or ChatGPT-User, the agent for user actions, can do. If you allow both OAI-SearchBot and GPTBot, OpenAI can use the results of one crawl for both purposes.

Identify real requests

Anyone can copy a user-agent string. OpenAI publishes the IP ranges of OAI-SearchBot in searchbot.json. Make sure that the IP address of a request is inside one of the listed prefixes. OpenAI’s crawler page does not describe a reverse DNS method.

The file has a creationTime field and a prefixes list with one ipv4Prefix value for each range. If jq is on your computer, this command prints the ranges:

curl -s https://openai.com/searchbot.json | jq -r '.prefixes[].ipv4Prefix'

If a firewall or CDN blocks unknown bots on your site, add these ranges to its allowlist. OpenAI recommends that you allow requests from its published IP ranges. The list changes from time to time, so update your allowlist when creationTime changes.

Block OAI-SearchBot on the server

robots.txt is a request. A server rule is a firm block. This nginx example returns HTTP 403 for the token:

if ($http_user_agent ~* "OAI-SearchBot") {
    return 403;
}

Put the example in the server block of your site configuration, then reload nginx. The rule stops only requests that carry the token. To block by address, use the ranges in searchbot.json.

What a block changes

After a block, ChatGPT search answers do not show your pages, except as navigational links. A block does not stop ChatGPT-User, so ChatGPT can still open a page when a user asks for it. A block also says nothing about training. Only a GPTBot rule does that.