# YandexBot: Yandex Crawler User Agent and robots.txt

> YandexBot is the main Yandex search crawler. User-agent strings, robots.txt tokens, crawl rate, and the DNS check for real requests.

YandexBot is the main indexing robot of the Yandex search engine, and it follows robots.txt. If you want your pages in Yandex search, allow `YandexBot`. If you block it, your pages cannot enter the search index.

## User-agent strings

The [robots help page](https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots) lists this string for the main indexing robot:

```
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
```

Your logs can also show the browser version that the robot used to crawl the page. The help page gives this example:

```
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268
```

The browser version can change. The help page recommends that you do not include the version when you search your logs for this user agent.

A second YandexBot variant finds the other addresses (mirrors) of a site:

```
Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)
```

The same page lists many other robots, for example `YandexImages`, `YandexVideo`, `YandexMetrika` and `YaDirectFetcher`. Some of them do not follow the general robots.txt rules.

## robots.txt tokens

The [User-agent directive page](https://yandex.com/support/webmaster/en/robot-workings/user-agent) gives these rules:

- The robots look for a `User-agent` line that contains `Yandex` (case does not matter) or `*`.
- If a `User-agent: Yandex` group exists, the robots do not use the `User-agent: *` group.
- A `User-agent: YandexBot` group applies only to the main indexing robot.

The help page gives this example:

```
User-agent: YandexBot # will be used only by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except the main indexing bot

User-agent: * # will not be used by Yandex bots
Disallow: /cgi-bin
```

If you want to block only the main indexing robot, use this group:

```
User-agent: YandexBot
Disallow: /
```

Some robots do not obey `User-agent: Yandex` or `User-agent: *`. If you want to control one of these robots, name its exact token, for example `User-agent: YandexCalendar`.

YandexBot stopped using the `Crawl-delay` directive on February 22, 2018. If you want to set a crawl rate, use the crawl rate setting in Yandex Webmaster ([Crawl-delay page](https://yandex.com/support/webmaster/en/robot-workings/crawl-delay)).

## AI answers

AI answers in Yandex Search use pages that YandexBot already indexed. A separate token, `YandexAdditional`, keeps a page out of these answers but leaves it in search. The [YandexGPT guide](/ai-search-engine/yandex-gpt/) explains that opt-out.

## Verify a YandexBot request

Other bots can copy the YandexBot user agent. The help page says that the robots use the networks AS13238, AS208722 and AS212066, and IP addresses that change often. For this reason, no list of IP addresses is published. The help page recommends a reverse DNS check:

1. Find the IP address of the request in your server logs.
2. Do a reverse DNS lookup of the IP address to get the host name.
3. Make sure that the host name ends in `yandex.ru`, `yandex.net` or `yandex.com`.
4. Do a forward DNS lookup of that host name.
5. Make sure that the result is the same IP address as in step 1.

```
host 203.0.113.10
host
```

Yandex also has an IP address check tool for its indexing robots. The help page prefers the DNS check to IP rules, because the internal networks change.

## Block YandexBot on the server

robots.txt is the documented way to control YandexBot. If you block YandexBot with a firewall instead, use the DNS check above, not only the user agent. A robots.txt block does not remove a page from search. Yandex says that [pages restricted in robots.txt can still take part in Yandex search](https://yandex.com/support/webmaster/en/controlling-robot/robots-txt). To remove a page, use the `noindex` directive in the page or in an HTTP header.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/yandexbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)