Google Gemini: Web Access, Google-Extended and Sources
Gemini is the AI assistant from Google. Gemini Apps can answer with content from the Google Search index, which Googlebot builds. If you do not want Gemini to use your pages for answers or for model training, add a Google-Extended rule to robots.txt. This rule does not remove your site from Google Search.
How Gemini gets web content
Gemini Apps use grounding. Grounding means that Google gives the model content from the Google Search index at prompt time. Google describes this in its documentation for Google-Extended. There is no separate Gemini crawler for this content. Googlebot crawls the pages, and Google Search keeps them in its index.
Some Gemini products also get a page when a user asks for it. Google lists them as user-triggered fetchers:
| User agent token | What it does, according to Google |
|---|---|
Google-GeminiNotebook |
Gets the URLs that Gemini Notebook users add as sources for their projects. |
Google-NotebookLM |
The former token of the same fetcher. Google gave August 2026 as the end of its support. |
Google-Agent |
Used by agents on Google infrastructure that go to web pages and do actions when a user asks. |
Each full user agent string contains its token, for example compatible; Google-Agent;. The full strings for desktop and mobile are on the Google page.
Block or allow Gemini with Google-Extended
Google-Extended is a robots.txt token. It has no HTTP user agent string of its own. Google crawls with its usual user agents and reads the token as a control. According to Google, the token controls these uses of your content:
- Training of future Gemini models for Gemini Apps and the Vertex AI API for Gemini.
- Grounding in Gemini Apps.
- Grounding with Google Search on Vertex AI.
If you want to stop these uses, add this group to your robots.txt file:
User-agent: Google-Extended
Disallow: /Google says that Google-Extended does not change whether your site is in Google Search. It is also not a ranking signal. The Google-Extended guide has more examples.
Google-Extended does not control AI Overviews or AI Mode. Google’s page on AI features and your website says that robots.txt rules for Googlebot are the control for these Search features. The preview controls nosnippet, data-nosnippet, max-snippet and noindex also apply. The AI Overviews guide explains them.
User-triggered fetchers and robots.txt
Google says that user-triggered fetchers generally ignore robots.txt, because a user asked for the fetch. So a Google-Extended rule does not stop Google-GeminiNotebook or Google-Agent. If you want to refuse these requests, add a rule for the token in your web server or firewall.
Other clients can send a Google user agent string. Google publishes IP ranges and reverse DNS names to identify real requests:
Google-Agentuses the IP ranges inuser-triggered-agents.json.- The host names of user-triggered fetchers end in
gae.googleusercontent.com, or they start withgoogle-proxy-and end ingoogle.com.
If you block a fetcher by IP address, use these JSON files and not a fixed list.
How Gemini shows sources
A Gemini response can have a Sources button at the bottom or inside the text. Google explains it on its help page about related sources. The button opens a side panel with links. The links can go to public websites, files that the user uploaded, or Google Workspace documents and emails.
If Gemini quotes a large amount of text from a web page, the sources list links to that page. Some responses have no Sources button, because Gemini gave no links for them.