Rogerbot: Moz Site Audit Crawler and robots.txt
Rogerbot is the Moz Pro site audit crawler, separate from the Dotbot link crawler. How to allow, block or slow it in robots.txt and how to identify it.
Rogerbot is the crawler that Moz uses for site audits in Moz Pro, the SEO software of Moz. It is not a search engine crawler and it does not build the Moz link index. Moz says that rogerbot obeys robots.txt, so you can allow, slow or block it there.
What rogerbot does
The Moz rogerbot page says that rogerbot gets the data for these Moz Pro reports:
- Site Crawl in a Moz Pro Campaign
- On-Demand Crawl
- Page Optimisation report
- On-Page Grader
Rogerbot and Dotbot
Moz runs two crawlers with different jobs:
| Bot | robots.txt token | What it is for |
|---|---|---|
| rogerbot | rogerbot |
Site audits for Moz Pro Campaigns and the reports above |
| Dotbot | dotbot |
The Moz Link Index for Link Explorer, the Links section of Moz Pro and the Moz Links API |
The Moz Dotbot page describes the second bot. If you block rogerbot, Dotbot can still crawl your site for the link index. Write a group for each bot that you want to control. See also the Dotbot guide.
User agent
Moz tells site owners to identify its crawler by the name rogerbot in the user agent. The Moz help page does not give the full user agent string.
Allow, block or slow rogerbot
Moz gives these robots.txt examples.
If you want to block rogerbot from the whole site, add this group:
User-agent: rogerbot
Disallow: /If you want to block only some URLs, use a pattern. This group blocks every URL that contains /marketplace/:
User-agent: rogerbot
Disallow: */marketplace/*Rogerbot supports Allow. This group lets rogerbot crawl one subfolder and blocks the rest of the folder:
User-agent: rogerbot
Allow: /category/illustration/
Disallow: /category/If you want to slow rogerbot down, set a delay in seconds. With this group, rogerbot requests one page every 10 seconds, which is at most 8,640 pages a day:
User-agent: rogerbot
Crawl-delay: 10Moz does not recommend a delay of more than 30 seconds. With a longer delay, rogerbot can fail to finish the crawl.
If you want to block all other bots and allow only rogerbot, Moz says to put the rogerbot group above the group for all bots:
User-agent: rogerbot
Disallow:
User-agent: *
Disallow: /Moz also says that rogerbot can have trouble crawling a site when robots.txt does not load or returns an error.
IP addresses
Moz does not use a fixed IP address or IP range for rogerbot. It uses many IP addresses that change for each crawl. So you cannot allow or block rogerbot by IP address.
If you want to block it at the server or firewall, match rogerbot in the user agent. Any client can send that name, so a match does not prove that a request came from Moz.