# How to Find and Block AI Crawlers on Your Site > Find AI crawlers in your access log, match their IP addresses to operator lists, and control them with robots.txt and firewall rules. This page shows how to see which AI crawlers visit your site and how to control them. Find them in your access log, make sure that the IP addresses are real, then write a robots.txt group for each bot. For what the bot types are, see [AI crawler bots: training, search and user fetchers](/ai-crawler-bot/introduction-to-ai-crawler-bots/). ## Find AI crawlers in your access log Your web server writes each request to an access log, with the user agent of the client. Search the log for the names of known AI bots. The path in these examples is the default nginx log on Debian and Ubuntu. Use the log path of your server. ```bash grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot" /var/log/nginx/access.log ``` To count the requests for each bot, run this command: ```bash grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot" /var/log/nginx/access.log | sort | uniq -c | sort -rn ``` If your site is behind a CDN, the CDN logs show the same data. Cloudflare marks the bots that it [verified](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/). A user agent alone does not prove who sent a request. Any client can send the name GPTBot. ## Make sure that a request is real Operators publish IP lists, reverse DNS host names, or both. If the IP address of a request is not on the list of the operator, treat the request as unknown. | Operator | Bots | IP list | |---|---|---| | [OpenAI](https://developers.openai.com/api/docs/bots) | GPTBot, OAI-SearchBot, ChatGPT-User | [gptbot.json](https://openai.com/gptbot.json), [searchbot.json](https://openai.com/searchbot.json), [chatgpt-user.json](https://openai.com/chatgpt-user.json) | | [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | ClaudeBot, Claude-SearchBot, Claude-User | [bots.json](https://claude.com/crawling/bots.json) | | [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | PerplexityBot, Perplexity-User | [perplexitybot.json](https://www.perplexity.ai/perplexitybot.json), [perplexity-user.json](https://www.perplexity.ai/perplexity-user.json) | | [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) | Googlebot and other common crawlers | [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json) | | [Common Crawl](https://commoncrawl.org/ccbot) | CCBot | [ccbot.json](https://index.commoncrawl.org/ccbot.json) | Google and Common Crawl also support reverse DNS. Google documents these steps: 1. Run a reverse DNS lookup on the IP address from your log with the `host` command. 2. Make sure that the host name ends in `googlebot.com`, `google.com` or `googleusercontent.com`. 3. Run a forward DNS lookup on that host name with the `host` command. 4. Make sure that the result is the IP address from your log. For CCBot, the host name ends in `crawl.commoncrawl.org`. ## Control crawlers with robots.txt robots.txt is a text file at the top level of your site, at `/robots.txt`. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) is the standard for it. These facts from the standard affect AI crawler rules: - A crawler matches its token to the `User-agent` lines without regard to case. - If more than one group matches a crawler, the crawler combines their rules. - The standard tells crawlers not to use a cached copy of the file for more than 24 hours. A change can take up to a day to apply. - The rules "are not a form of access authorization". A bot can ignore them. This example blocks two training crawlers, the Google-Extended token and CCBot. It allows the AI search crawlers: ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / ``` ## What a robots.txt rule changes Each rule changes only the bot that it names. The operators document these effects: - OpenAI says that a site can allow OAI-SearchBot to appear in ChatGPT search results and disallow GPTBot. Disallowing GPTBot means that the content is not to be used to train its foundation models. - Google says Google-Extended does not affect inclusion or ranking in Google Search. - OpenAI says robots.txt rules do not always apply to ChatGPT-User, because a user starts the action. Perplexity says Perplexity-User generally ignores robots.txt. - Anthropic says that its bots honor robots.txt and support the non-standard `Crawl-delay` rule. ## Block at the server or firewall If a bot ignores robots.txt, block it on your server, CDN or firewall by user agent or by IP range. A user agent block stops only the bots that send their real name. An IP block needs the current list from the operator. Anthropic says that an IP block can stop its bots from reading your robots.txt file. For this reason, an IP block is not a reliable way to opt out of Anthropic crawling. ## A bot name that no operator documents Your access log can show a bot name that no operator documents, for example `ia_gensim` or `ISSCyborg`. A name alone does not tell you who sent the request. Run a reverse DNS lookup on the IP address, and compare the address with the operator lists above. If no list has the address and no known operator owns the host name, treat the request as an unknown bot. You can add a robots.txt group with the exact name from your log, but an unknown bot can ignore it. If the requests continue, block the IP address or the user agent on your server or firewall. ## Guides for single bots - OpenAI: [GPTBot](/ai-crawler-bot/gptbot/), [OAI-SearchBot](/ai-crawler-bot/oai-searchbot/), [ChatGPT-User](/ai-crawler-bot/chatgpt-user/) - Anthropic: [ClaudeBot](/ai-crawler-bot/claudebot/), [Claude-SearchBot](/ai-crawler-bot/claude-searchbot/), [Claude-User](/ai-crawler-bot/claude-user/) - Perplexity: [PerplexityBot](/ai-crawler-bot/perplexitybot/), [Perplexity-User](/ai-crawler-bot/perplexity-user/) - Google: [Googlebot](/ai-crawler-bot/googlebot/), [Google-Extended](/ai-crawler-bot/google-extended/) - Common Crawl: [CCBot](/ai-crawler-bot/ccbot/) - Bots with little or no documentation: [GrokBot](/ai-search-engine/grok/), [Kangaroo Bot](/ai-crawler-bot/kangaroo-bot/) --- [View the canonical page](https://aichatwatch.com/ai-crawler-bot/introduction-to-ai-crawlers/) ยท [Browse llms.txt](https://aichatwatch.com/llms.txt)