Google-InspectionTool: User Agent for Search Testing Tools

Google-InspectionTool is the crawler that Google’s Search testing tools use, for example URL Inspection in Search Console and the Rich Results Test. Rules for it affect only these tools, not Google Search. If you test your pages with these tools, let it crawl the pages that you test.

User-agent strings

Google’s list of common crawlers gives a desktop and a mobile string.

Desktop:

Mozilla/5.0 (compatible; Google-InspectionTool/1.0)

Mobile:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)

Chrome/W.X.Y.Z is a placeholder for the current Chrome version. If you match the mobile string in logs or server rules, use a wildcard for the version.

What it is used for

Google says that rules for Google-InspectionTool affect Search testing tools such as the Rich Results Test and URL Inspection in Search Console. Google also says these rules have no effect on Google Search or other products.

You see this user agent in your logs when one of these tools fetches a page from your site.

robots.txt tokens

Google lists two tokens for this crawler: Google-InspectionTool and Googlebot. A token is the name in a robots.txt User-agent line. The crawler uses the group with the most specific matching token (robots.txt specification).

This has two results:

1. If your robots.txt has no Google-InspectionTool group, the crawler follows your Googlebot group.
2. If your robots.txt has a Google-InspectionTool group, the crawler follows only that group.

Google lists this crawler among its common crawlers, which always obey robots.txt when they crawl automatically.

If you want to keep the testing tools out of a folder, add this group:

User-agent: Google-InspectionTool
Disallow: /private/

This rule does not change how Googlebot crawls the folder for Google Search.

Verify requests

Google’s common crawlers, including Google-InspectionTool and Googlebot, generally crawl from the same published IP ranges. Google’s verification guide gives a DNS method and an IP list method. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

For an automatic method, compare the IP address with the ranges in common-crawlers.json.