MojeekBot: Mojeek Crawler User Agent and robots.txt

MojeekBot is the web crawler of Mojeek, a UK search engine that builds its own index. It obeys robots.txt and the robots meta tag. If you want your pages in Mojeek search, allow MojeekBot. If you do not, disallow it in robots.txt.

User agent and robots.txt token

Mojeek’s MojeekBot page uses the token MojeekBot. The page does not print a full user-agent string. To find the crawler in your logs, search for MojeekBot.

Mojeek says that MojeekBot obeys the first robots.txt group whose User-agent line contains “MojeekBot”. If no group matches, it obeys the first * group. A group for Mojeek alone does not match, because it does not contain the full token.

Block or limit MojeekBot in robots.txt

To block the full site, add this group:

User-agent: MojeekBot
Disallow: /

MojeekBot does not fetch a URL that contains a disallowed string. Mojeek gives this example: the rule below blocks both /private/ and /private.html.

User-agent: MojeekBot
Disallow: /private

Mojeek says that MojeekBot is meant to request no more than one page from your site in the same second. MojeekBot does not support the Crawl-delay line.

Robots meta tag

Mojeek says that its engine obeys the noindex, nocache and nofollow values of the robots meta tag. With noindex, MojeekBot fetches the page but does not add it to the search index:

<meta name="robots" content="noindex">

The two controls have different effects. A robots.txt disallow stops the fetch. A noindex tag lets the fetch happen but keeps the page out of search results. MojeekBot must be able to fetch the page to see the tag. The robots meta tag guide explains the tag values.

Identify real requests

Any client can send a user agent that contains “MojeekBot”. Mojeek documents a two-step DNS method:

1. Do a reverse DNS lookup on the IP address of the request. Make sure that the name ends in mojeek.com, for example crawl-5-102-173-71.mojeek.com.
2. Do a forward DNS lookup on that name. Make sure that it returns the original IP address.

If the forward lookup returns a different address, the request did not come from MojeekBot.

You can also compare the IP address with Mojeek’s list of MojeekBot IP addresses. Mojeek says that it can update the list at any time, so download it again often.