YandexBot: Yandex Crawler User Agent and robots.txt

YandexBot is the main indexing robot of the Yandex search engine, and it follows robots.txt. If you want your pages in Yandex search, allow YandexBot. If you block it, your pages cannot enter the search index.

User-agent strings

The robots help page lists this string for the main indexing robot:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Your logs can also show the browser version that the robot used to crawl the page. The help page gives this example:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268

The browser version can change. The help page recommends that you do not include the version when you search your logs for this user agent.

A second YandexBot variant finds the other addresses (mirrors) of a site:

Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)

The same page lists many other robots, for example YandexImages, YandexVideo, YandexMetrika and YaDirectFetcher. Some of them do not follow the general robots.txt rules.

robots.txt tokens

The User-agent directive page gives these rules:

• The robots look for a User-agent line that contains Yandex (case does not matter) or *.
• If a User-agent: Yandex group exists, the robots do not use the User-agent: * group.
• A User-agent: YandexBot group applies only to the main indexing robot.

The help page gives this example:

User-agent: YandexBot # will be used only by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except the main indexing bot

User-agent: * # will not be used by Yandex bots
Disallow: /cgi-bin

If you want to block only the main indexing robot, use this group:

User-agent: YandexBot
Disallow: /

Some robots do not obey User-agent: Yandex or User-agent: *. If you want to control one of these robots, name its exact token, for example User-agent: YandexCalendar.

YandexBot stopped using the Crawl-delay directive on February 22, 2018. If you want to set a crawl rate, use the crawl rate setting in Yandex Webmaster (Crawl-delay page).

AI answers

AI answers in Yandex Search use pages that YandexBot already indexed. A separate token, YandexAdditional, keeps a page out of these answers but leaves it in search. The YandexGPT guide explains that opt-out.

Verify a YandexBot request

Other bots can copy the YandexBot user agent. The help page says that the robots use the networks AS13238, AS208722 and AS212066, and IP addresses that change often. For this reason, no list of IP addresses is published. The help page recommends a reverse DNS check:

1. Find the IP address of the request in your server logs.
2. Do a reverse DNS lookup of the IP address to get the host name.
3. Make sure that the host name ends in yandex.ru, yandex.net or yandex.com.
4. Do a forward DNS lookup of that host name.
5. Make sure that the result is the same IP address as in step 1.

host 203.0.113.10
host <hostname-from-the-first-command>

Yandex also has an IP address check tool for its indexing robots. The help page prefers the DNS check to IP rules, because the internal networks change.

Block YandexBot on the server

robots.txt is the documented way to control YandexBot. If you block YandexBot with a firewall instead, use the DNS check above, not only the user agent. A robots.txt block does not remove a page from search. Yandex says that pages restricted in robots.txt can still take part in Yandex search. To remove a page, use the noindex directive in the page or in an HTTP header.