PetalBot: Huawei Petal Search Crawler and robots.txt

PetalBot is the crawler of Petal Search, the search engine from Huawei. It builds the index for Petal Search and for content suggestions in Huawei Assistant and AI Search. Huawei says that PetalBot obeys robots.txt, so you can block or limit it with a PetalBot group.

What PetalBot does

Huawei’s PetalBot page says that PetalBot visits PC and mobile websites and builds an index database. Users can then find your content in Petal Search. Huawei also uses the index for content suggestions in Huawei Assistant and AI Search services, which Petal Search powers.

User-agent strings

A user agent is the client name in the User-Agent HTTP header. Huawei lists two strings:

Crawler	Full user agent
PetalBot (PC)	Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
PetalBot (mobile)	Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

The robots.txt token is PetalBot.

Block or limit PetalBot in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Huawei states: “PetalBot complies with the Internet robots protocol.” If you want to block PetalBot on the whole site, add this group:

User-agent: PetalBot
Disallow: /

If you want to block only some paths, list them. Huawei gives this example:

User-agent: PetalBot
Allow: /w/api/
Disallow: /trap/

If PetalBot sends too many requests, Huawei asks site owners to write to [email protected]. Huawei says that it adjusts the crawl rate to server capacity, site quality and site updates.

What a block changes

Huawei warns that a block makes the pages on your site unsearchable in Petal Search. A block does not remove pages that are already in the index at once. Huawei says that removal from the index can take several months. If the removal is urgent, Huawei asks you to write to [email protected].

Verify requests

Anyone can copy a user-agent string. Huawei recommends a reverse DNS lookup (a query that finds the hostname of an IP address). Do these steps on a host with the host command:

1. Run host <IP address> on the IP address from your logs.
2. Make sure that the hostname ends in petalsearch.com.
3. Run host on that hostname.
4. Make sure that the result is the same IP address as in your logs.

The example on Huawei’s page uses hostnames in the form petalbot-<IP address with dashes>.petalsearch.com. The PetalBot page gives no IP list.

Related Huawei crawler

Third-party bot lists also name PanguBot as a Huawei crawler. Huawei does not document it on the PetalBot page. A PetalBot group in robots.txt does not apply to PanguBot.