The robots meta tag tells search engine crawlers whether they can index a page, follow its links and show a snippet of it. Put it in the page <head>, or send the same rules in an X-Robots-Tag HTTP header. Do not block the page in robots.txt, because then crawlers cannot read the tag.

Syntax

<meta name="robots" content="noindex, nofollow">

The name robots applies to all crawlers. If you want a rule for one crawler only, use the name of that crawler:

<meta name="googlebot" content="nosnippet">

You can put several rules in one tag, separated by commas, or in separate tags. Google says that the name, the content and the rules are case-insensitive. The tag is not part of the HTML standard. MDN calls it a de facto standard for crawlers.

Rules that Google supports

Google lists these rules in its robots meta tag specification:

Rule	What Google does
`noindex`	Does not show the page in search results
`nofollow`	Does not follow the links on the page
`none`	Same as `noindex, nofollow`
`nosnippet`	Shows no text snippet or video preview for the page
`max-snippet:[number]`	Limits the text snippet to that number of characters. `0` means no snippet, `-1` means no limit
`max-image-preview:[setting]`	Sets the largest image preview: `none`, `standard` or `large`
`max-video-preview:[number]`	Limits a video preview to that number of seconds. `-1` means no limit
`noimageindex`	Does not index the images on the page
`notranslate`	Does not offer a translation of the title link and snippet
`indexifembedded`	Together with `noindex`, lets Google index the content when another page embeds it in an iframe
`unavailable_after:[date]`	Stops showing the page after the date. Use a format such as ISO 8601 or RFC 822
`all`	No restrictions. This is the default

If two rules conflict, Google uses the more restrictive rule. For example, nosnippet wins over max-snippet:50. Google ignores noarchive, nocache and nositelinkssearchbox.

For Google Search, the tag accepts two crawler names: googlebot for text results and googlebot-news for news results. Google ignores other names.

Google AI Overviews and AI Mode

Google says that nosnippet applies to all forms of its search results, including AI Overviews and AI Mode. Google also says that nosnippet stops the content from being used as a direct input for AI Overviews and AI Mode.

max-snippet also applies to AI Overviews and AI Mode. Google says that this limit does not apply where a publisher separately gave permission to use the content.

If you want to hide only part of a page from snippets, put the data-nosnippet attribute on a span, div or section element:

<p>This text can appear in a snippet. <span data-nosnippet>This text cannot.</span></p>

Google’s AI features page names nosnippet, data-nosnippet, max-snippet and noindex as the controls for what Search shows. Google-Extended is a robots.txt token, not a meta tag rule. Google says that it limits AI training and grounding in some of its other systems.

Rules that Bing supports

In an April 2020 post, Bing announced support for max-snippet, max-image-preview and max-video-preview. The post says that Bing still supports nosnippet. Bing reads these rules in the meta tag and in the X-Robots-Tag header.

In a September 2023 post, Bing gave two rules a meaning for Bing Chat:

nocache: Bing Chat can use the page, but answers show only the URL, the title and the snippet. Only those parts can be used to train Microsoft’s generative AI foundation models.
noarchive: Bing Chat does not use the page in answers and does not link to it. Bing does not use the content to train Microsoft’s generative AI foundation models.

If a page has both rules, Bing treats it as nocache. Bing says that pages with either rule still appear in Bing search results. Microsoft later renamed Bing Chat to Copilot. The post does not say how the rules apply to Copilot.

<meta name="robots" content="noarchive, nocache">

Rules that Applebot supports

Apple says that Applebot supports noindex, nosnippet, nofollow, none and all in robots meta tags (Apple support). Applebot also reads the X-Robots-Tag header. Apple describes only a robots.txt control for Applebot-Extended, the token for Apple AI training.

X-Robots-Tag header

If the file is not HTML, for example a PDF or an image, send the rules in an HTTP response header:

X-Robots-Tag: noindex
X-Robots-Tag: googlebot: nofollow

A rule without a crawler name applies to all crawlers. Google recommends the header for PDF, video and image files.

noai and noimageai

DeviantArt added the noai and noimageai directives in November 2022 to mark art that must not be used to train AI (TechCrunch). DeviantArt’s terms tell third parties to exclude this content from AI training data sets. The directives are not in Google’s list of supported rules. If you want to block an AI crawler, use the robots.txt token that its operator documents, for example GPTBot.

What the robots meta tag does not do

The robots meta tag does not do these things:

It does not stop crawling. A crawler must fetch the page to read the tag. Google says that if robots.txt blocks the page, Google does not see the rules and ignores them.
It does not protect content. MDN says that only cooperative robots follow these rules.
It does not remove a page at once. MDN says that noindex takes effect only after the robot visits the page again.
In Google Search, noarchive and nocache have no effect.

The meta tags cheat sheet lists the other tags that search engines document.