# Google-CloudVertexBot: Vertex AI Crawler and robots.txt

> Google-CloudVertexBot crawls sites for Vertex AI search (Agent Search) data stores. User agent, robots.txt tokens, sitemaps and verification.

Google-CloudVertexBot crawls websites for Google Cloud's Vertex AI search product, which Google is renaming to Agent Search. Google says site owners request these crawls to build Vertex AI Agents, and that rules for this bot have no effect on Google Search. If you use Agent Search on your own site, allow the bot. If you do not, you can block it.

## User-agent string

Google's [list of common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-cloudvertexbot) gives only a substring of the user-agent string:

```text
Google-CloudVertexBot
```

The Google Cloud page [Prepare data for ingesting](https://docs.cloud.google.com/generative-ai-app-builder/docs/prepare-data#website) shows a full example:

```text
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.7559.132 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search)
```

Google's crawler list gives only the substring. If you match this bot in logs or server rules, match `Google-CloudVertexBot`.

## What it is used for

A Google Cloud customer can create a data store for website search in Agent Search. Google-CloudVertexBot crawls and indexes the pages of that data store. The Google Cloud documentation says the bot needs to crawl paywalled content too, if the data store includes it.

For advanced website indexing, the customer must [verify the domains](https://docs.cloud.google.com/generative-ai-app-builder/docs/domain-verification) in the data store. A customer who does not own a domain can send an association request instead. The domain owner can approve the request in Google Search Console.

## robots.txt tokens

Google lists two tokens for this bot: `Google-CloudVertexBot` and `Googlebot`. A token is the name in a robots.txt `User-agent` line. The bot uses the group with the most specific matching token ([robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)).

1. If your robots.txt has a `Google-CloudVertexBot` group, the bot follows only that group.
2. If it has no such group, the bot follows your `Googlebot` group.

If you want to block the bot from your whole site, add this group to `/robots.txt`:

```text
User-agent: Google-CloudVertexBot
Disallow: /
```

If you use Agent Search on your site, allow it instead:

```text
User-agent: Google-CloudVertexBot
Allow: /
```

## Sitemaps need Googlebot

Agent Search crawls pages with Google-CloudVertexBot, but it fetches the sitemap file with [Googlebot](/ai-crawler-bot/googlebot/). If your robots.txt, CDN or firewall allows Google-CloudVertexBot but blocks Googlebot, Agent Search still crawls your pages. But it cannot read the sitemap, so it does not index or refresh URLs through the sitemap.

## What blocking does not change

Google says rules for `Google-CloudVertexBot` have no effect on Google Search or other products. A block does not change [Google-Extended](/ai-crawler-bot/google-extended/) either. Google-Extended is the separate control for Gemini training and for grounding in Gemini Apps and Vertex AI.

## Verify requests

Google's common crawlers, including this bot, generally use the IP ranges in [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json). Their hostnames match `crawl-***-***-***-***.googlebot.com` or `geo-crawl-***-***-***-***.geo.googlebot.com`. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname matches one of the two patterns.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

Google describes these steps in its [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/google-cloudvertexbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)