GPTBot: OpenAI Crawler, User Agent and robots.txt
GPTBot user agent, robots.txt rules, IP ranges and what a block changes. Blocking GPTBot does not remove a site from ChatGPT search.
GPTBot is the OpenAI crawler that collects content that can be used to train OpenAI’s generative AI foundation models. If you do not want your content used for this training, disallow GPTBot in robots.txt. This block does not remove your site from ChatGPT search, which uses a separate bot, OAI-SearchBot.
User-agent string
OpenAI’s bots page gives this example string and says that the version number can change:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotWhen GPTBot fetches robots.txt files, OpenAI can add a robots.txt marker to the string. The marker helps you find robots.txt requests in logs that do not show paths.
The robots.txt token is GPTBot. A token is the name in a robots.txt User-agent line.
What GPTBot is for
OpenAI says it uses GPTBot to make its generative AI foundation models more useful and safe. GPTBot crawls content that can be used in training these models. OpenAI says that a disallow rule for GPTBot tells OpenAI not to use the site’s content in that training.
Block GPTBot in robots.txt
If you want to keep GPTBot out of your whole site, add this group to /robots.txt:
User-agent: GPTBot
Disallow: /If you want to keep it out of one folder only, disallow that path:
User-agent: GPTBot
Disallow: /members/OpenAI says each bot setting is independent of the others. For example, you can stay in ChatGPT search and opt out of training:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /If you allow both bots, OpenAI can use the results of one crawl for both uses.
What blocking GPTBot does not change
A GPTBot rule does not control ChatGPT search. OAI-SearchBot controls whether your site appears in ChatGPT search answers. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, but they can still appear as navigational links.
A GPTBot rule also does not stop ChatGPT-User. That agent visits pages when a ChatGPT user asks for them.
The bots page does not say whether a block removes content that GPTBot collected before the block.
Verify GPTBot requests
OpenAI publishes the GPTBot IP ranges at openai.com/gptbot.json. Make sure that the IP address of a request is in this list before you treat it as GPTBot. The bots page does not describe a reverse DNS method.
If you want to block GPTBot in a firewall or CDN, you can match the GPTBot user agent. Other clients can copy the string. Only the published IP ranges show that a request comes from OpenAI.
OpenAI bots compared
| User agent | Purpose | robots.txt | IP list |
|---|---|---|---|
GPTBot |
Content for training foundation models | Controlled by the GPTBot token |
gptbot.json |
OAI-SearchBot |
Shows sites in ChatGPT search features | Controlled by the OAI-SearchBot token |
searchbot.json |
ChatGPT-User |
Visits pages for user actions in ChatGPT and custom GPTs | OpenAI says rules do not always apply | chatgpt-user.json |
OAI-AdsBot |
Reviews landing pages of ads submitted to ChatGPT | Not stated | adsbot.json |
OpenAI says ChatGPT-User is not used to decide whether content appears in search. It also says OAI-AdsBot visits only pages submitted as ads, and that its data is not used to train foundation models. For search results, OpenAI says a robots.txt change can take about 24 hours to apply.
Some logs and robots.txt templates also show the name OAI-Research. OpenAI does not document a bot with this name, so a robots.txt group for it has no documented effect.