# DotBot: Moz Link Index Crawler and robots.txt Rules

> DotBot is the crawler that builds the Moz Link Index. Its robots.txt token, how to block or slow it, and how it differs from Rogerbot.

DotBot is the web crawler of Moz, a company that makes SEO software. It collects link data for the Moz Link Index, which feeds Link Explorer, the Moz API and the Links section of Moz Pro campaigns. Moz says that DotBot always respects robots.txt. If you want to block or slow it, add a `dotbot` group to your robots.txt file.

## What DotBot collects

Moz describes the crawler on its [DotBot help page](https://moz.com/help/moz-procedures/crawlers/dotbot). DotBot gathers web data for the Moz Link Index. The data appears in Link Explorer, the Moz Links API and the Links section of a Moz Pro campaign.

Moz says that a machine learning model selects the pages that DotBot crawls. Moz trains the model to start with high-value links ([How we index the web](https://moz.com/help/link-explorer/getting-started/how-we-index)).

## DotBot and Rogerbot

Moz runs a second crawler. [Rogerbot](/ai-crawler-bot/rogerbot/) crawls sites for Moz Pro Site Crawl. If a site has a Moz Pro Campaign, Rogerbot crawls it every week. DotBot does not do site audits.

Moz's Performance Metrics report uses neither crawler. Google Lighthouse collects that data with the user agent `Chrome-Lighthouse`.

## User agent and robots.txt token

Moz gives `dotbot` as the name to use in robots.txt. The help page does not publish a full user agent string, IP ranges or a reverse DNS host name. So you cannot confirm from Moz documents that a request with this name came from Moz.

## Block or slow DotBot

robots.txt is a text file at the root of a site that gives rules to crawlers ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)). If you want to block DotBot from your whole site, add this group:

```
User-agent: dotbot
Disallow: /
```

If you want to block only some folders, list them:

```
User-agent: dotbot
Disallow: /admin/
Disallow: /scripts/
Disallow: /images/
```

If you want DotBot to crawl more slowly, add a crawl delay. Crawl-delay is not part of RFC 9309, but Moz documents it for DotBot:

```
User-agent: dotbot
Crawl-delay: 10
```

## When DotBot sees changes

DotBot reads your robots.txt file only the first time it finds your site during a new index crawl. If you change the file later, DotBot sees the change the next time that Moz finds links to your site. So a new rule does not apply at once.

## What a block changes

Moz says that if DotBot finds a DotBot exclusion in robots.txt or a noindex meta tag, Moz does not show that data in its index. The block affects only Moz data. A `dotbot` group does not change how Googlebot, Bingbot or other crawlers treat your site. For other SEO crawlers, see [AhrefsBot](/ai-crawler-bot/ahrefsbot/) and [SemrushBot](/ai-crawler-bot/semrushbot/).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/dotbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)