# MojeekBot: Mojeek Crawler User Agent and robots.txt

> MojeekBot crawls for the Mojeek search engine. robots.txt token, noindex support, crawl rate and how to identify real requests.

MojeekBot is the web crawler of [Mojeek](/ai-search-engine/mojeek/), a UK search engine that builds its own index. It obeys robots.txt and the robots meta tag. If you want your pages in Mojeek search, allow `MojeekBot`. If you do not, disallow it in robots.txt.

## User agent and robots.txt token

Mojeek's [MojeekBot page](https://www.mojeek.com/bot.html) uses the token `MojeekBot`. The page does not print a full user-agent string. To find the crawler in your logs, search for `MojeekBot`.

Mojeek says that MojeekBot obeys the first robots.txt group whose User-agent line contains "MojeekBot". If no group matches, it obeys the first `*` group. A group for `Mojeek` alone does not match, because it does not contain the full token.

## Block or limit MojeekBot in robots.txt

To block the full site, add this group:

```
User-agent: MojeekBot
Disallow: /
```

MojeekBot does not fetch a URL that contains a disallowed string. Mojeek gives this example: the rule below blocks both `/private/` and `/private.html`.

```
User-agent: MojeekBot
Disallow: /private
```

Mojeek says that MojeekBot is meant to request no more than one page from your site in the same second. MojeekBot does not support the Crawl-delay line.

## Robots meta tag

Mojeek says that its engine obeys the `noindex`, `nocache` and `nofollow` values of the robots meta tag. With `noindex`, MojeekBot fetches the page but does not add it to the search index:

```

```

The two controls have different effects. A robots.txt disallow stops the fetch. A `noindex` tag lets the fetch happen but keeps the page out of search results. MojeekBot must be able to fetch the page to see the tag. The [robots meta tag guide](/html-tags/meta-robots/) explains the tag values.

## Identify real requests

Any client can send a user agent that contains "MojeekBot". Mojeek documents a two-step DNS method:

1. Do a reverse DNS lookup on the IP address of the request. Make sure that the name ends in `mojeek.com`, for example `crawl-5-102-173-71.mojeek.com`.
2. Do a forward DNS lookup on that name. Make sure that it returns the original IP address.

If the forward lookup returns a different address, the request did not come from MojeekBot.

You can also compare the IP address with Mojeek's [list of MojeekBot IP addresses](https://www.mojeek.com/mojeekbot.json). Mojeek says that it can update the list at any time, so download it again often.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/mojeekbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)