# Storebot-Google: Google Shopping Crawler and robots.txt

> Storebot-Google crawls product, cart and checkout pages for Google Shopping. User agents, robots.txt token, verification and what a block changes.

Storebot-Google is the Google crawler for Google Shopping. It reads product pages and checkout flows to collect price, shipping and availability data. If you sell through Google Merchant Center, let it crawl your shop. A robots.txt rule for `Storebot-Google` affects all Google Shopping surfaces.

## What Storebot-Google does

Google describes the crawler in the Merchant Center help article [About the Google StoreBot crawler](https://support.google.com/merchants/answer/13294660). It goes through product detail pages, cart pages and checkout pages. Google says that it tries to go through the full checkout flow without a purchase. In your logs, it can show as `StoreBot-Google` or as a user named "Google StoreBot".

The crawler collects data such as:

- Product and shipping prices
- Online and local availability
- Whether coupons are valid
- Shipping times and countries
- Payment methods
- Contact information

Google compares this data with the information that merchants give in Merchant Center. If you add structured data to your pages or enter data in Merchant Center, that data takes precedence over what the crawler extracts.

## User-agent strings

Google's [list of common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-storebot) gives two strings.

Desktop:

```text
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36
```

Mobile:

```text
Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36
```

`Chrome/W.X.Y.Z` is a placeholder for the Chrome version. The version changes over time.

## robots.txt rules for Storebot-Google

The robots.txt token is `Storebot-Google`. A token is the name in a robots.txt `User-agent` line. Google says that its common crawlers always obey robots.txt rules when they crawl automatically.

If you want to keep the crawler out of one folder, add a group like this to `/robots.txt`:

```text
User-agent: Storebot-Google
Disallow: /private/
```

Google lists only this one token for the crawler. So Storebot-Google does not follow a `User-agent: Googlebot` group. If your file has no `Storebot-Google` group, the crawler follows the `User-agent: *` group.

## What blocking changes

Google says that rules for `Storebot-Google` affect all surfaces of Google Shopping, for example the Shopping tab in Google Search. The rules do not apply to [Googlebot](/ai-crawler-bot/googlebot/), which crawls for Google Search.

Local inventory needs crawler access too. Google's [help article about blocked in-store product pages](https://support.google.com/merchants/answer/13484511) covers this case. If the crawler cannot reach an in-store product page, Merchant Center shows a warning or a suspension. This affects local inventory ads and free local listings. The article lists these causes:

- A robots.txt file that blocks Google crawlers
- Bot detection software with a block list that contains the StoreBot user agent
- Firewall or CDN rules that return errors only to bots
- Allow lists with only the IP addresses of Google Search crawlers
- Slow landing pages that time out

## Verify requests

Google says that its common crawlers generally use the IP ranges in [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json). The Merchant Center article warns that StoreBot can use other IP addresses than the Google Search crawlers. It recommends a reverse DNS lookup instead of a fixed IP allow list. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname ends in `googlebot.com`.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

Google's [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) gives the hostname patterns for common crawlers: `crawl-***-***-***-***.googlebot.com` and `geo-crawl-***-***-***-***.geo.googlebot.com`.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/storebot-google/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)