DotBot: Moz Link Index Crawler and robots.txt Rules
DotBot is the crawler that builds the Moz Link Index. Its robots.txt token, how to block or slow it, and how it differs from Rogerbot.
DotBot is the web crawler of Moz, a company that makes SEO software. It collects link data for the Moz Link Index, which feeds Link Explorer, the Moz API and the Links section of Moz Pro campaigns. Moz says that DotBot always respects robots.txt. If you want to block or slow it, add a dotbot group to your robots.txt file.
What DotBot collects
Moz describes the crawler on its DotBot help page. DotBot gathers web data for the Moz Link Index. The data appears in Link Explorer, the Moz Links API and the Links section of a Moz Pro campaign.
Moz says that a machine learning model selects the pages that DotBot crawls. Moz trains the model to start with high-value links (How we index the web).
DotBot and Rogerbot
Moz runs a second crawler. Rogerbot crawls sites for Moz Pro Site Crawl. If a site has a Moz Pro Campaign, Rogerbot crawls it every week. DotBot does not do site audits.
Moz’s Performance Metrics report uses neither crawler. Google Lighthouse collects that data with the user agent Chrome-Lighthouse.
User agent and robots.txt token
Moz gives dotbot as the name to use in robots.txt. The help page does not publish a full user agent string, IP ranges or a reverse DNS host name. So you cannot confirm from Moz documents that a request with this name came from Moz.
Block or slow DotBot
robots.txt is a text file at the root of a site that gives rules to crawlers (RFC 9309). If you want to block DotBot from your whole site, add this group:
User-agent: dotbot
Disallow: /If you want to block only some folders, list them:
User-agent: dotbot
Disallow: /admin/
Disallow: /scripts/
Disallow: /images/If you want DotBot to crawl more slowly, add a crawl delay. Crawl-delay is not part of RFC 9309, but Moz documents it for DotBot:
User-agent: dotbot
Crawl-delay: 10When DotBot sees changes
DotBot reads your robots.txt file only the first time it finds your site during a new index crawl. If you change the file later, DotBot sees the change the next time that Moz finds links to your site. So a new rule does not apply at once.
What a block changes
Moz says that if DotBot finds a DotBot exclusion in robots.txt or a noindex meta tag, Moz does not show that data in its index. The block affects only Moz data. A dotbot group does not change how Googlebot, Bingbot or other crawlers treat your site. For other SEO crawlers, see AhrefsBot and SemrushBot.