Baiduspider: Baidu Crawler User Agents and robots.txt
Baiduspider crawls the web for Baidu search. User agent tokens per Baidu product, reverse DNS verification and robots.txt rules from Baidu's help pages.
Baiduspider is the crawler of Baidu, the Chinese search engine. It visits pages and adds them to the Baidu index. If you want visitors from Baidu search, allow it. If you block it in robots.txt, your pages leave Baidu search and the other search services that Baidu provides.
What Baiduspider does
Baidu answers common questions about the crawler in its FAQs of Baiduspider (English) and in a longer Chinese version. Baidu says that Baiduspider visits pages on the internet and builds the Baidu index, so that users can find your site in Baidu search.
These Baidu help pages do not mention AI training. They also do not name a separate AI crawler or an AI opt-out token.
User agent tokens by Baidu product
Baidu uses a different token for each product:
| Baidu product | User agent token |
|---|---|
| PC search | Baiduspider |
| Mobile search | Baiduspider |
| Image search | Baiduspider-image |
| Video search | Baiduspider-video |
| News search | Baiduspider-news |
| Baidu bookmarks | Baiduspider-favo |
| Baidu Union | Baiduspider-cpro |
| Business search | Baiduspider-ads |
The Chinese FAQ gives the full strings for web search. PC search:
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)Mobile search:
Mozilla/5.0 (Linux;u;Android 4.2.2;zh-cn;) AppleWebKit/534.46 (KHTML,like Gecko) Version/5.1 Mobile Safari/10600.6.3 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)Both strings contain Baiduspider/2.0. According to Baidu, the words Android and Mobile mark a request from the mobile crawler.
Verify Baiduspider requests
Baidu warns that other bots can pretend to be Baiduspider. Baidu does not publish an IP list on these pages. It tells you to use DNS:
- Do a reverse DNS lookup on the IP address. The host name must end in
.baidu.comor.baidu.jp. - Do a forward DNS lookup on that host name. Make sure that it returns the same IP address.
This is Baidu’s Linux example:
$ host 123.125.66.120
120.66.125.123.in-addr.arpa domain name pointer baiduspider-123-125-66-120.crawl.baidu.com.
$ host baiduspider-123-125-66-120.crawl.baidu.com
baiduspider-123-125-66-120.crawl.baidu.com has address 123.125.66.120On Windows, use nslookup. On macOS, use dig -x for the reverse lookup.
Block or limit Baiduspider in robots.txt
Baidu states that Baiduspider follows robots.txt. If you want to block all Baidu crawlers, block the main token:
User-agent: Baiduspider
Disallow: /If you want to block Baidu but let the image crawler into one folder, use Baidu’s example:
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider-image
Allow: /image/Baidu documents these limits:
Baiduspider-videodoes not support robots.txt rules at this time, according to the English FAQ.Baiduspider-cproandBaiduspider-adsdo not follow robots.txt. Their pages do not go into the index. They act on agreements with Baidu customers. For problems withBaiduspider-cpro, Baidu gives the address [email protected].- A new robots.txt rule usually takes effect in about 48 hours. Pages already in the index can take several months to leave the search results.
Baidu also follows robots meta tags in the page. For an urgent removal or a crawl load problem, Baidu asks you to send feedback through its webmaster platform. Include your server logs.
What a robots.txt block does in Baidu search
The Baiduspider help page says that Baidu does not crawl, index or show the content of pages that robots.txt blocks. A blocked page can still show in results if other sites link to it. In that case, the text in the result comes from the other sites.
Baidu also reads meta tags with the name Baiduspider. This tag stops Baidu from showing a cached copy of the page:
<meta name="Baiduspider" content="noarchive">Baidu says that it still indexes the page and shows a snippet in the results.