Google-CloudVertexBot: Vertex AI Crawler and robots.txt

Google-CloudVertexBot crawls websites for Google Cloud’s Vertex AI search product, which Google is renaming to Agent Search. Google says site owners request these crawls to build Vertex AI Agents, and that rules for this bot have no effect on Google Search. If you use Agent Search on your own site, allow the bot. If you do not, you can block it.

User-agent string

Google’s list of common crawlers gives only a substring of the user-agent string:

Google-CloudVertexBot

The Google Cloud page Prepare data for ingesting shows a full example:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.7559.132 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search)

Google’s crawler list gives only the substring. If you match this bot in logs or server rules, match Google-CloudVertexBot.

What it is used for

A Google Cloud customer can create a data store for website search in Agent Search. Google-CloudVertexBot crawls and indexes the pages of that data store. The Google Cloud documentation says the bot needs to crawl paywalled content too, if the data store includes it.

For advanced website indexing, the customer must verify the domains in the data store. A customer who does not own a domain can send an association request instead. The domain owner can approve the request in Google Search Console.

robots.txt tokens

Google lists two tokens for this bot: Google-CloudVertexBot and Googlebot. A token is the name in a robots.txt User-agent line. The bot uses the group with the most specific matching token (robots.txt specification).

1. If your robots.txt has a Google-CloudVertexBot group, the bot follows only that group.
2. If it has no such group, the bot follows your Googlebot group.

If you want to block the bot from your whole site, add this group to /robots.txt:

User-agent: Google-CloudVertexBot
Disallow: /

If you use Agent Search on your site, allow it instead:

User-agent: Google-CloudVertexBot
Allow: /

Sitemaps need Googlebot

Agent Search crawls pages with Google-CloudVertexBot, but it fetches the sitemap file with Googlebot. If your robots.txt, CDN or firewall allows Google-CloudVertexBot but blocks Googlebot, Agent Search still crawls your pages. But it cannot read the sitemap, so it does not index or refresh URLs through the sitemap.

What blocking does not change

Google says rules for Google-CloudVertexBot have no effect on Google Search or other products. A block does not change Google-Extended either. Google-Extended is the separate control for Gemini training and for grounding in Gemini Apps and Vertex AI.

Verify requests

Google’s common crawlers, including this bot, generally use the IP ranges in common-crawlers.json. Their hostnames match crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname matches one of the two patterns.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

Google describes these steps in its verification guide.