# Rogerbot: Moz Site Audit Crawler and robots.txt

> Rogerbot is the Moz Pro site audit crawler, separate from the Dotbot link crawler. How to allow, block or slow it in robots.txt and how to identify it.

Rogerbot is the crawler that Moz uses for site audits in Moz Pro, the SEO software of Moz. It is not a search engine crawler and it does not build the Moz link index. Moz says that rogerbot obeys robots.txt, so you can allow, slow or block it there.

## What rogerbot does

The [Moz rogerbot page](https://moz.com/help/moz-procedures/crawlers/rogerbot) says that rogerbot gets the data for these Moz Pro reports:

- Site Crawl in a Moz Pro Campaign
- On-Demand Crawl
- Page Optimisation report
- On-Page Grader

## Rogerbot and Dotbot

Moz runs two crawlers with different jobs:

| Bot | robots.txt token | What it is for |
|---|---|---|
| rogerbot | `rogerbot` | Site audits for Moz Pro Campaigns and the reports above |
| Dotbot | `dotbot` | The Moz Link Index for Link Explorer, the Links section of Moz Pro and the Moz Links API |

The [Moz Dotbot page](https://moz.com/help/moz-procedures/crawlers/dotbot) describes the second bot. If you block rogerbot, Dotbot can still crawl your site for the link index. Write a group for each bot that you want to control. See also the [Dotbot guide](/ai-crawler-bot/dotbot/).

## User agent

Moz tells site owners to identify its crawler by the name `rogerbot` in the user agent. The Moz help page does not give the full user agent string.

## Allow, block or slow rogerbot

Moz gives these robots.txt examples.

If you want to block rogerbot from the whole site, add this group:

```
User-agent: rogerbot
Disallow: /
```

If you want to block only some URLs, use a pattern. This group blocks every URL that contains `/marketplace/`:

```
User-agent: rogerbot
Disallow: */marketplace/*
```

Rogerbot supports `Allow`. This group lets rogerbot crawl one subfolder and blocks the rest of the folder:

```
User-agent: rogerbot
Allow: /category/illustration/
Disallow: /category/
```

If you want to slow rogerbot down, set a delay in seconds. With this group, rogerbot requests one page every 10 seconds, which is at most 8,640 pages a day:

```
User-agent: rogerbot
Crawl-delay: 10
```

Moz does not recommend a delay of more than 30 seconds. With a longer delay, rogerbot can fail to finish the crawl.

If you want to block all other bots and allow only rogerbot, Moz says to put the rogerbot group above the group for all bots:

```
User-agent: rogerbot
Disallow:

User-agent: *
Disallow: /
```

Moz also says that rogerbot can have trouble crawling a site when robots.txt does not load or returns an error.

## IP addresses

Moz does not use a fixed IP address or IP range for rogerbot. It uses many IP addresses that change for each crawl. So you cannot allow or block rogerbot by IP address.

If you want to block it at the server or firewall, match `rogerbot` in the user agent. Any client can send that name, so a match does not prove that a request came from Moz.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/rogerbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)