CCBot: Common Crawl Crawler, User Agent and Blocking

CCBot is the crawler of Common Crawl, a non-profit that publishes a free archive of web crawl data. Many AI developers use this archive as training data. If you do not want your pages in future Common Crawl archives, disallow CCBot in robots.txt.

What CCBot collects

The Common Crawl FAQ describes the organization as a 501(c)(3) non-profit. It gives a copy of the internet to researchers, companies and individuals at no cost. The data is stored on Amazon S3 for bulk download. The archive is a sample of the web. Common Crawl says that it does not usually archive an entire website, but a randomly selected subset of it.

A 2024 Mozilla Foundation report calls Common Crawl one of the most important sources of training data for generative AI. The same report says that the archive is not the entire web and not a representative sample of it.

User-agent string

The user agent is the client name in the User-Agent HTTP header. The CCBot page gives this string:

CCBot/2.0 (https://commoncrawl.org/faq/)

The older bot used this string:

CCBot/1.0 (+https://commoncrawl.org/bot.html)

The robots.txt token is CCBot. Common Crawl says that it can increase the version number in the future.

How CCBot crawls

Per the FAQ, CCBot reads robots.txt first. If a page is allowed, it fetches the page with an HTTP GET request. It supports HTTP/1.1 and HTTP/2, and IPv4 and IPv6. It does not run JavaScript and does not use cookies. It follows up to four redirects, or five for robots.txt, in line with RFC 9309.

CCBot slows down when your server returns HTTP 429 or a 5xx status. It reads sitemaps that your robots.txt announces, and it obeys nofollow on links.

Block CCBot in robots.txt

robots.txt is a text file at the site root with rules for crawlers. If you want to stop CCBot, add this group:

User-agent: CCBot
Disallow: /

Common Crawl says that it continues to read your robots.txt from time to time to find changes.

If you want fewer requests instead of a block, use Crawl-delay. Common Crawl obeys it. This example asks for at most one request every 2 seconds:

User-agent: CCBot
Crawl-delay: 2

Common Crawl also keeps an Opt-Out Ledger, a public list of the opt-out requests that it received. For opt-out requests, the post gives the address [email protected].

What a block does not change

A robots.txt rule stops future crawls. Common Crawl does not say that the rule removes your pages from archives that it already published.

A block for CCBot does not block other AI crawlers. Each operator has its own token, for example GPTBot for OpenAI and ClaudeBot for Anthropic.

Verify requests

Common Crawl warns that some crawlers falsely identify themselves as CCBot. The real CCBot uses dedicated IP ranges with reverse DNS (a lookup from an IP address to a host name). Reverse DNS is not available over IPv6 yet.

To make sure that an IPv4 address from your logs belongs to CCBot, run a reverse lookup:

host 18.97.14.84

The host name must end with .crawl.commoncrawl.org. Then run a forward lookup on that name and make sure that it returns the same IP address:

host 18-97-14-84.crawl.commoncrawl.org

Common Crawl also publishes its IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json.

Find your pages in the archive

You can look up URLs in the Common Crawl index at index.commoncrawl.org. The FAQ says that this endpoint has strict rate limits. If you send too many requests, your IP address can be blocked for a time.