# CCBot: Common Crawl Crawler, User Agent and Blocking

> CCBot collects pages for the Common Crawl web archive, a common AI training source. User agent, IP ranges, reverse DNS and robots.txt rules.

CCBot is the crawler of Common Crawl, a non-profit that publishes a free archive of web crawl data. Many AI developers use this archive as training data. If you do not want your pages in future Common Crawl archives, disallow `CCBot` in robots.txt.

## What CCBot collects

The [Common Crawl FAQ](https://commoncrawl.org/faq) describes the organization as a 501(c)(3) non-profit. It gives a copy of the internet to researchers, companies and individuals at no cost. The data is stored on Amazon S3 for bulk download. The archive is a sample of the web. Common Crawl says that it does not usually archive an entire website, but a randomly selected subset of it.

A 2024 [Mozilla Foundation report](https://www.mozillafoundation.org/research/library/generative-ai-training-data/) calls Common Crawl one of the most important sources of training data for generative AI. The same report says that the archive is not the entire web and not a representative sample of it.

## User-agent string

The user agent is the client name in the User-Agent HTTP header. The [CCBot page](https://commoncrawl.org/ccbot) gives this string:

```
CCBot/2.0 (https://commoncrawl.org/faq/)
```

The older bot used this string:

```
CCBot/1.0 (+https://commoncrawl.org/bot.html)
```

The robots.txt token is `CCBot`. Common Crawl says that it can increase the version number in the future.

## How CCBot crawls

Per the FAQ, CCBot reads robots.txt first. If a page is allowed, it fetches the page with an HTTP GET request. It supports HTTP/1.1 and HTTP/2, and IPv4 and IPv6. It does not run JavaScript and does not use cookies. It follows up to four redirects, or five for robots.txt, in line with [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html).

CCBot slows down when your server returns HTTP 429 or a 5xx status. It reads sitemaps that your robots.txt announces, and it obeys `nofollow` on links.

## Block CCBot in robots.txt

robots.txt is a text file at the site root with rules for crawlers. If you want to stop CCBot, add this group:

```
User-agent: CCBot
Disallow: /
```

Common Crawl says that it continues to read your robots.txt from time to time to find changes.

If you want fewer requests instead of a block, use `Crawl-delay`. Common Crawl obeys it. This example asks for at most one request every 2 seconds:

```
User-agent: CCBot
Crawl-delay: 2
```

Common Crawl also keeps an [Opt-Out Ledger](https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry), a public list of the opt-out requests that it received. For opt-out requests, the post gives the address info@commoncrawl.org.

## What a block does not change

A robots.txt rule stops future crawls. Common Crawl does not say that the rule removes your pages from archives that it already published.

A block for CCBot does not block other AI crawlers. Each operator has its own token, for example [GPTBot](/ai-crawler-bot/gptbot/) for OpenAI and [ClaudeBot](/ai-crawler-bot/claudebot/) for Anthropic.

## Verify requests

Common Crawl warns that some crawlers falsely identify themselves as CCBot. The real CCBot uses dedicated IP ranges with reverse DNS (a lookup from an IP address to a host name). Reverse DNS is not available over IPv6 yet.

To make sure that an IPv4 address from your logs belongs to CCBot, run a reverse lookup:

```
host 18.97.14.84
```

The host name must end with `.crawl.commoncrawl.org`. Then run a forward lookup on that name and make sure that it returns the same IP address:

```
host 18-97-14-84.crawl.commoncrawl.org
```

Common Crawl also publishes its IPv4 and IPv6 ranges as JSON at [index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json).

## Find your pages in the archive

You can look up URLs in the Common Crawl index at [index.commoncrawl.org](https://index.commoncrawl.org). The FAQ says that this endpoint has strict rate limits. If you send too many requests, your IP address can be blocked for a time.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/ccbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)