Meta-ExternalAgent: Meta AI Crawler and robots.txt
Meta-ExternalAgent crawls the web to train Meta AI models and index content. User agent, robots.txt rules and what a block changes.
Meta-ExternalAgent is a web crawler that Meta runs. Meta says it collects web content to train foundation AI models and to improve Meta products by indexing content. If you do not want Meta to collect your pages this way, disallow meta-externalagent in your robots.txt file.
What the crawler is for
Meta lists Meta-ExternalAgent on its web crawlers page. The page gives two example uses: training foundation AI models, and improving products by indexing content directly.
Meta-ExternalAgent does not make link previews. Previews of links that people share in Meta apps come from FacebookExternalHit, a different Meta crawler.
User-agent string
Meta says that the user agent in your logs has one of two forms. The short form is:
meta-externalagent/1.1The long form adds a link to Meta’s crawler documentation in parentheses after the version number.
The robots.txt token is meta-externalagent. Under RFC 9309, crawlers match the token without regard to case, so Meta-ExternalAgent also works.
Block Meta-ExternalAgent in robots.txt
Meta tells site owners to add a disallow rule for the crawler that they want to block. To block the full site, add this group to /robots.txt:
User-agent: meta-externalagent
Disallow: /If you want to block only one folder, use a narrower rule. This example follows the format of Meta’s own example:
User-agent: meta-externalagent
Allow: /
Disallow: /private/Meta says that its crawlers can cache robots.txt for up to 24 hours. A change can take that long to have an effect.
Meta names two of its crawlers that do not always follow robots.txt: Meta-ExternalFetcher and FacebookExternalHit. Meta does not say this about Meta-ExternalAgent.
What a block changes
A robots.txt group applies only to the crawler that it names. A block on meta-externalagent does not block the other Meta crawlers in the table below. Link previews in Facebook, Instagram and Messenger continue, because FacebookExternalHit makes them.
Meta says that when you allow Meta-WebIndexer in robots.txt, it helps Meta AI cite and link to your content in its answers. If you want citations in Meta AI, do not block meta-webindexer. Meta does not say whether content that Meta-WebIndexer collects is also used for training.
Other Meta crawlers
This table summarizes Meta’s crawler page.
| User agent | Use, as Meta describes it | robots.txt notes |
|---|---|---|
facebookexternalhit/1.1 |
Title, description and image for links shared in Meta apps | Can skip robots.txt for security or integrity checks |
meta-webindexer/1.1 |
Improve Meta AI search results | Allowing it helps Meta AI cite and link to you |
meta-externalads/1.1 |
Improve advertising and other business products | Included in Meta’s robots.txt example |
meta-externalagent/1.1 |
Train foundation AI models, index content for products | Block with a disallow rule |
meta-externalfetcher/1.1 |
Fetch single links when a user asks | Can skip robots.txt |
The Meta-ExternalFetcher guide explains how to handle the user-initiated fetcher.
Identify real requests
Meta’s crawler page does not publish an IP address list or a reverse DNS method for Meta-ExternalAgent. Any client can send this user-agent string, so the string alone does not prove that a request came from Meta.
If a client with this user agent ignores your robots.txt group, block that user agent at your web server, CDN or firewall. Meta’s crawler page gives [email protected] as the contact for questions about its crawlers.