Meta AI: Chat, Web Search, Crawlers and Opt-Out
Meta AI chat and search use Bing, Google and the meta-webindexer crawler. How Meta AI cites pages and how to allow or block Meta crawlers.
Meta AI is the assistant from Meta in Facebook, Instagram, WhatsApp, Messenger and on meta.ai. For current information, it uses web search results, and Meta runs a crawler named Meta-WebIndexer to improve Meta AI search and cite web pages. If you want Meta AI to cite your site, allow meta-webindexer in robots.txt. If you want to opt out, disallow it.
Where Meta AI gets web results
Meta has named three sources over time:
- In September 2023, Meta said that Meta AI gets real-time information through a search partnership with Bing (Meta newsroom).
- In April 2024, Meta added real-time results from Google next to Bing (Search Engine Journal).
- Meta now lists its own crawler, Meta-WebIndexer, on its web crawlers page. Meta says it crawls the web to improve the quality of Meta AI search results. Meta also says that allowing it helps Meta cite and link your content in Meta AI responses.
Meta does not document how it combines these sources today. It also does not say if Bing or Google page controls, such as the Bing noarchive tag, apply to Meta AI.
The Meta AI chatbot
Meta announced Meta AI on April 18, 2024, and built it with its Llama 3 model. The post says that people can get real-time information from across the web without leaving the app. The Meta AI terms say that Meta can share certain information with select partners, like search engines, to give relevant results. The terms do not name the partners. Meta also does not publish details about the format of citations in the chat.
Meta crawlers that relate to Meta AI
| robots.txt token | What Meta says it does | robots.txt |
|---|---|---|
meta-webindexer |
Improves Meta AI search results and helps Meta cite and link your content. | Disallow it to opt out. |
meta-externalagent |
Crawls for uses such as training foundation AI models, or improving products by indexing content directly. | Disallow it to opt out. |
meta-externalfetcher |
Fetches single links at a user’s request. | Can bypass robots.txt, because the user requested the fetch. |
facebookexternalhit |
Fetches link details when someone shares a link in a Meta app. | Can bypass robots.txt for security or integrity checks. |
Meta lists meta-webindexer/1.1 as a user agent string. Another listed form adds a link to the Meta crawler documentation after it. The other Meta crawlers use the same pattern, for example meta-externalagent/1.1.
Allow search, block training
This example lets Meta AI search crawl your site and blocks the crawler that Meta lists for AI training:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /If you want to opt out of Meta AI search, change the first group to Disallow: /. Meta says that its crawlers can cache robots.txt, so changes can take up to 24 hours.
If a user asks Meta AI about a link, meta-externalfetcher can fetch it even when robots.txt disallows it. If you must stop it, block its user agent on your web server, CDN or firewall. The block also stops Meta AI from reading the page for the person who asked.
Identify real Meta requests
The Meta crawler page does not publish IP ranges for these crawlers or a reverse DNS name. You can match only on the user agent, and any client can copy a user agent. Meta takes crawler questions at [email protected].