# Google Gemini: Web Access, Google-Extended and Sources

> How Gemini uses web pages: grounding from the Google Search index, the Google-Extended robots.txt token, user-triggered fetchers and sources.

Gemini is the AI assistant from Google. Gemini Apps can answer with content from the Google Search index, which Googlebot builds. If you do not want Gemini to use your pages for answers or for model training, add a `Google-Extended` rule to robots.txt. This rule does not remove your site from Google Search.

## How Gemini gets web content

Gemini Apps use grounding. Grounding means that Google gives the model content from the Google Search index at prompt time. Google describes this in its [documentation for Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers). There is no separate Gemini crawler for this content. [Googlebot](/ai-crawler-bot/googlebot/) crawls the pages, and Google Search keeps them in its index.

Some Gemini products also get a page when a user asks for it. Google lists them as [user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers):

| User agent token | What it does, according to Google |
|---|---|
| `Google-GeminiNotebook` | Gets the URLs that Gemini Notebook users add as sources for their projects. |
| `Google-NotebookLM` | The former token of the same fetcher. Google gave August 2026 as the end of its support. |
| `Google-Agent` | Used by agents on Google infrastructure that go to web pages and do actions when a user asks. |

Each full user agent string contains its token, for example `compatible; Google-Agent;`. The full strings for desktop and mobile are on the Google page.

## Block or allow Gemini with Google-Extended

`Google-Extended` is a robots.txt token. It has no HTTP user agent string of its own. Google crawls with its usual user agents and reads the token as a control. According to Google, the token controls these uses of your content:

- Training of future Gemini models for Gemini Apps and the Vertex AI API for Gemini.
- Grounding in Gemini Apps.
- Grounding with Google Search on Vertex AI.

If you want to stop these uses, add this group to your robots.txt file:

```
User-agent: Google-Extended
Disallow: /
```

Google says that Google-Extended does not change whether your site is in Google Search. It is also not a ranking signal. The [Google-Extended guide](/ai-crawler-bot/google-extended/) has more examples.

Google-Extended does not control AI Overviews or AI Mode. Google's page on [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) says that robots.txt rules for Googlebot are the control for these Search features. The preview controls `nosnippet`, `data-nosnippet`, `max-snippet` and `noindex` also apply. The [AI Overviews guide](/ai-search-engine/google-ai-overviews/) explains them.

## User-triggered fetchers and robots.txt

Google says that user-triggered fetchers generally ignore robots.txt, because a user asked for the fetch. So a `Google-Extended` rule does not stop `Google-GeminiNotebook` or `Google-Agent`. If you want to refuse these requests, add a rule for the token in your web server or firewall.

Other clients can send a Google user agent string. Google publishes [IP ranges and reverse DNS names](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) to identify real requests:

- `Google-Agent` uses the IP ranges in `user-triggered-agents.json`.
- The host names of user-triggered fetchers end in `gae.googleusercontent.com`, or they start with `google-proxy-` and end in `google.com`.

If you block a fetcher by IP address, use these JSON files and not a fixed list.

## How Gemini shows sources

A Gemini response can have a Sources button at the bottom or inside the text. Google explains it on its [help page about related sources](https://support.google.com/gemini/answer/14143489?hl=en). The button opens a side panel with links. The links can go to public websites, files that the user uploaded, or Google Workspace documents and emails.

If Gemini quotes a large amount of text from a web page, the sources list links to that page. Some responses have no Sources button, because Gemini gave no links for them.

---

[View the canonical page](https://aichatwatch.com/ai-chat-bot/google-gemini/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)