Google-Extended: robots.txt Token for Gemini Training

Google-Extended is a robots.txt token, not a separate crawler. It controls whether Google can use content it crawls from your site to train future Gemini models and to ground Gemini answers. It does not change your inclusion or ranking in Google Search, and it does not control AI Overviews or AI Mode.

What Google-Extended controls

Google describes Google-Extended in its list of common crawlers as a standalone product token. If you disallow it, Google applies your choice to two uses:

1. Training future generations of Gemini models that power Gemini Apps and the Vertex AI API for Gemini.
2. Grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Grounding is when Google gives the model content from the Google Search index at prompt time. Google says the purpose is better factuality and relevance.

Google announced Google-Extended on September 28, 2023. The announcement named Bard and the Vertex AI generative APIs. The current documentation names Gemini Apps and Vertex AI.

What it does not control

Google says Google-Extended does not affect whether your site is in Google Search. Google also says it is not a ranking signal.

AI Overviews and AI Mode are part of Google Search. For these features, Google’s AI features page names robots.txt rules for Googlebot as the control. To limit the text that Search shows from a page, use nosnippet, data-nosnippet, max-snippet or noindex. A Google-Extended rule does not remove your pages from AI Overviews.

No user-agent string

Google-Extended has no user-agent string of its own. Google crawls with its existing user-agent strings and uses the Google-Extended token only as a control. You will not see Google-Extended in your server logs.

For this reason, a server or firewall rule cannot separate Google-Extended from Search crawling. A rule that blocks Google’s crawler user agents also blocks Googlebot. Use robots.txt for this control.

Block Google-Extended in robots.txt

If you want to opt your whole site out of Gemini training and grounding, add this group to /robots.txt:

User-agent: Google-Extended
Disallow: /

If you want to opt out only one folder, disallow that path:

User-agent: Google-Extended
Disallow: /articles/

Keep your Googlebot rules as they are. The Google-Extended group does not change them.

Google lists only one token for this control: Google-Extended. For this reason, a User-agent: Googlebot group does not set your Google-Extended choice. Google caches robots.txt for up to 24 hours (robots.txt specification), so a change can take a day to apply.

Related Google tokens

Token	What it controls
Googlebot	Google Search, including AI Overviews and AI Mode
GoogleOther	Crawls by product teams and internal research, no specific product
Google-CloudVertexBot	Crawls that site owners request for Vertex AI Agents

For other companies’ AI crawlers, see the robots.txt guide for AI crawlers.