MJ12bot: Majestic Crawler User Agent and robots.txt

MJ12bot is the web crawler of Majestic, a UK company that runs a backlink index. It maps the links between websites. It is not an AI crawler. MJ12bot obeys robots.txt, so if you do not want it, add an MJ12bot group to robots.txt instead of an IP block.

What MJ12bot collects

Majestic’s MJ12bot page says that the crawler maps link relationships between websites. Majestic adds this link graph to its backlink index, which people search with the Site Explorer tool on majestic.com. Majestic says that MJ12bot does not currently cache web content or personal data.

MJ12bot is a distributed crawler. Volunteers with broadband connections run it as part of a community project. For this reason, requests come from many IP addresses around the world.

User agent and robots.txt token

Majestic’s page names the bot MJ12bot. It lists v1.4.8, from April 2017, as the current version. The page does not print a full user-agent string. To find the bot in your logs, search for the token MJ12bot. The robots.txt token is also MJ12bot.

Block MJ12bot in robots.txt

To block the full site, Majestic gives this group:

User-agent: MJ12bot
Disallow: /

Majestic asks site owners not to block the bot by IP address in .htaccess. The bot does not use consecutive IP blocks, so an IP block does not stop it reliably.

Make sure that MJ12bot can download your robots.txt file. If the bot cannot get the file at all, it assumes that it can crawl the site. Majestic also says that some fetch failures, such as 403 Forbidden on robots.txt, count as a full disallow.

Majestic says that rel="nofollow" on a link does not stop the bot from visiting the target page. Use robots.txt for that.

Slow MJ12bot down

If you want fewer requests instead of a full block, add a Crawl-Delay value in seconds:

User-agent: MJ12bot
Crawl-Delay: 5

MJ12bot supports Crawl-Delay values up to 20 seconds. It rounds higher values down to 20. Majestic says that more than one MJ12bot can crawl a site at the same time.

If robots.txt has an MJ12bot group, the bot uses that group instead of the * group. Copy any Crawl-Delay line from the * group into the MJ12bot group.

MJ12bot also supports these robots.txt extensions:

• Wildcard patterns in Disallow lines.
• Allow lines that override Disallow lines when the Allow path is longer.
• Redirects to the same site when it fetches robots.txt.

Identify real requests

Majestic cannot limit MJ12bot to a fixed set of IP addresses. Instead, Majestic can send a private ident string with every request to your site. The string comes in a CRAWLER-IDENT header or in the user agent. To set it up, send your site details to [email protected].

Without this string, the user agent alone does not prove that a request came from Majestic.

What a block changes

A block stops MJ12bot from reading the links on your pages. Links to your site from other websites are on pages that MJ12bot can still crawl.

Other link-index crawlers, such as AhrefsBot and SemrushBot, use their own robots.txt tokens. A group for MJ12bot does not apply to them.