# Meta-ExternalFetcher: User Agent and How to Block It

> Meta-ExternalFetcher fetches single links when a user asks a Meta product. It can bypass robots.txt. User agent and server blocking.

Meta-ExternalFetcher is a Meta user agent that fetches single links when a person asks a Meta product to open them. Meta says that it does not always follow robots.txt, because a user requested the fetch. If you must stop it, block its user agent at your web server, CDN or firewall.

## What Meta-ExternalFetcher does

Meta lists Meta-ExternalFetcher on its [web crawlers page](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers). Meta says that it fetches individual links at a user's request. Meta also says that it supports product functions such as the evaluation and improvement of agentic AI. One example that Meta gives is an AI that goes through websites to complete tasks for users.

Meta-ExternalFetcher does not make link previews. Previews of links that people share in Facebook, Instagram or Messenger come from [FacebookExternalHit](/ai-crawler-bot/facebookexternalhit/).

## User-agent string

Meta says that the user agent in your logs has one of two forms. The short form is:

```
meta-externalfetcher/1.1
```

The long form adds a link to Meta's crawler documentation in parentheses after the version number. The robots.txt token is `meta-externalfetcher`.

## robots.txt and Meta-ExternalFetcher

Meta says that this crawler can bypass robots.txt rules, because it does fetches that users requested. A robots.txt group for `meta-externalfetcher` states your preference, but Meta does not promise to obey it:

```
User-agent: meta-externalfetcher
Disallow: /
```

Meta says that its crawlers can cache robots.txt for up to 24 hours.

## Block Meta-ExternalFetcher at the server

If you must stop these requests, reject them by user agent at the web server, CDN or firewall. This nginx example returns HTTP 403 to each request whose user agent contains the token:

```
if ($http_user_agent ~* "meta-externalfetcher") {
    return 403;
}
```

Put the rule in the `server` block of the nginx configuration for your site.

A block also stops the Meta product from reading your page for the person who asked for it. Meta's crawler page does not publish an IP address list for this user agent. A server rule that matches the user agent does not stop a client that sends a different user agent.

## Related Meta crawlers

Each Meta crawler has its own robots.txt token. A rule for one token does not apply to the others.

| User agent | Use, as Meta describes it | robots.txt notes |
|---|---|---|
| `meta-externalfetcher/1.1` | Fetch single links when a user asks | Can bypass robots.txt |
| `meta-externalagent/1.1` | Train foundation AI models, index content for products | Block with a disallow rule |
| `meta-webindexer/1.1` | Improve Meta AI search results | Allowing it helps Meta AI cite and link to you |
| `facebookexternalhit/1.1` | Link previews in Meta apps | Can skip robots.txt for security or integrity checks |

The [Meta-ExternalAgent guide](/ai-crawler-bot/meta-externalagent/) lists all five crawlers on Meta's page.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/meta-externalfetcher/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)