Storebot-Google: Google Shopping Crawler and robots.txt

Storebot-Google is the Google crawler for Google Shopping. It reads product pages and checkout flows to collect price, shipping and availability data. If you sell through Google Merchant Center, let it crawl your shop. A robots.txt rule for Storebot-Google affects all Google Shopping surfaces.

What Storebot-Google does

Google describes the crawler in the Merchant Center help article About the Google StoreBot crawler. It goes through product detail pages, cart pages and checkout pages. Google says that it tries to go through the full checkout flow without a purchase. In your logs, it can show as StoreBot-Google or as a user named “Google StoreBot”.

The crawler collects data such as:

• Product and shipping prices
• Online and local availability
• Whether coupons are valid
• Shipping times and countries
• Payment methods
• Contact information

Google compares this data with the information that merchants give in Merchant Center. If you add structured data to your pages or enter data in Merchant Center, that data takes precedence over what the crawler extracts.

User-agent strings

Google’s list of common crawlers gives two strings.

Desktop:

Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36

Mobile:

Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36

Chrome/W.X.Y.Z is a placeholder for the Chrome version. The version changes over time.

robots.txt rules for Storebot-Google

The robots.txt token is Storebot-Google. A token is the name in a robots.txt User-agent line. Google says that its common crawlers always obey robots.txt rules when they crawl automatically.

If you want to keep the crawler out of one folder, add a group like this to /robots.txt:

User-agent: Storebot-Google
Disallow: /private/

Google lists only this one token for the crawler. So Storebot-Google does not follow a User-agent: Googlebot group. If your file has no Storebot-Google group, the crawler follows the User-agent: * group.

What blocking changes

Google says that rules for Storebot-Google affect all surfaces of Google Shopping, for example the Shopping tab in Google Search. The rules do not apply to Googlebot, which crawls for Google Search.

Local inventory needs crawler access too. Google’s help article about blocked in-store product pages covers this case. If the crawler cannot reach an in-store product page, Merchant Center shows a warning or a suspension. This affects local inventory ads and free local listings. The article lists these causes:

• A robots.txt file that blocks Google crawlers
• Bot detection software with a block list that contains the StoreBot user agent
• Firewall or CDN rules that return errors only to bots
• Allow lists with only the IP addresses of Google Search crawlers
• Slow landing pages that time out

Verify requests

Google says that its common crawlers generally use the IP ranges in common-crawlers.json. The Merchant Center article warns that StoreBot can use other IP addresses than the Google Search crawlers. It recommends a reverse DNS lookup instead of a fixed IP allow list. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname ends in googlebot.com.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

Google’s verification guide gives the hostname patterns for common crawlers: crawl-***-***-***-***.googlebot.com and geo-crawl-***-***-***-***.geo.googlebot.com.