Googlebot: Google Search Crawler User Agent and robots.txt

Googlebot is the crawler that finds and fetches pages for Google Search. A crawler is a program that downloads web pages automatically. If you want your pages in Google Search, including AI Overviews and AI Mode, let Googlebot crawl them.

User-agent strings

Google’s Googlebot page names two types: Googlebot Smartphone and Googlebot Desktop. For most sites, Google indexes the mobile version of a page. So most requests come from Googlebot Smartphone.

Googlebot Smartphone:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Google’s list of common crawlers also shows two rare forms:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot/2.1 (+http://www.google.com/bot.html)

Chrome/W.X.Y.Z is a placeholder for the Chrome version that Googlebot uses. The version changes over time. If you match these strings in logs or server rules, use a wildcard for the version.

robots.txt rules for Googlebot

Both types obey the token Googlebot. A token is the name in a robots.txt User-agent line. You cannot target only the smartphone crawler or only the desktop crawler. Google says its common crawlers always obey robots.txt when they crawl automatically.

If you want to keep Googlebot out of one folder, add this group to /robots.txt:

User-agent: Googlebot
Disallow: /private/

Google uses only one group for each crawler: the group with the most specific matching user agent (robots.txt specification). Googlebot-Image, Googlebot-Video, Googlebot-News, Google-InspectionTool and Google-CloudVertexBot also accept the Googlebot token. If your file has no group for their own token, they follow the Googlebot group. Google caches robots.txt for up to 24 hours, so a change can take a day to apply.

What blocking Googlebot changes

Google says that rules for Googlebot affect Google Search, Discover, all Search features, Google Images, Video mode and Google News. Robots.txt rules for Googlebot are also the control for AI Overviews and AI Mode (AI features and your website).

A robots.txt block stops crawling. It does not remove the URL from results. Indexing is when Google stores a page so that Search can show it. If you want a page out of Google Search, use a noindex rule. Do not block that page in robots.txt, because Googlebot must fetch the page to see the rule. To limit the text that Search shows, use nosnippet, data-nosnippet or max-snippet (robots meta tag).

If you want to stay in Search but keep your content out of Gemini training and grounding, allow Googlebot and disallow Google-Extended.

Crawl rate and file size

Google says Googlebot visits most sites no more than once every few seconds on average. For Search, Googlebot reads the first 2 MB of a supported file type and the first 64 MB of a PDF. If your server cannot keep up, you can reduce the crawl rate.

Verify Googlebot requests

Other bots often copy the Googlebot user-agent string. Google gives two methods to verify its crawlers. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For an automatic method, compare the IP address with the ranges in common-crawlers.json.

If you block or rate-limit bots in a firewall or CDN, use these IP ranges or the DNS method, not the user agent alone. Google’s crawler overview warns that wrong HTTP status codes can change how your site appears in Google products.

Other Google crawlers

Token	Used for
Googlebot-Image	Google Images and image features in Search
Googlebot-Video	Video features in Search
Googlebot-News	Google News (no separate user-agent string)
Storebot-Google	Google Shopping
GoogleOther	Product teams and internal research, no specific product
Google-Extended	Control token for Gemini training and grounding

Google’s Feedfetcher is a separate user-triggered fetcher for RSS and Atom feeds.