facebookexternalhit: Meta Link Preview Crawler

facebookexternalhit is the Meta crawler that fetches a page when someone shares a link to it on Facebook, Instagram or Messenger. It reads the title, description and thumbnail image for the link preview. If you want correct previews, let it fetch your pages. Put your Open Graph tags in the first 1 MB of the HTML. If you block it, Meta cannot show a preview for your links.

What it does

Meta describes the crawler on its Meta Web Crawlers page. Its main purpose is to crawl an app or website that someone shared on one of Meta’s apps, such as Facebook, Instagram or Messenger. The user can paste the link or share it with a Facebook social plugin. The crawler gathers, caches and displays the title, description and thumbnail image of the page.

User agent strings

Meta says that the user agent in your logs will be similar to one of these:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1

robots.txt behavior

Meta says that you set your preferences for its crawlers in robots.txt. To block a crawler, you add a disallow rule for it. Meta also says that facebookexternalhit can bypass robots.txt when it does security or integrity checks, for example checks for malware.

If you do not want Meta to fetch your pages for previews, add this group:

User-agent: facebookexternalhit
Disallow: /

Meta says that a change can take up to 24 hours, because its crawlers can cache robots.txt for that long.

Crawler requirements

Meta lists these requirements for the crawler:

1. Use gzip and deflate encodings on your server.
2. Put all Open Graph properties in the first 1 MB of the page. Meta cuts off the rest.
3. Make sure that the crawler can get the content within a few seconds. If it cannot, Facebook cannot show the content.
4. Return all required properties within the bytes that the Range header asks for, or ignore that header.
5. Add the user agent strings or the IP addresses of the crawler to your allow list. Meta calls IP addresses the more secure option.

For the tags themselves, see Open Graph meta tags and og:image.

Test and refresh a preview

Meta refreshes the link data of a page about every 30 days (How to Change the Metadata for an Existing Link). If you changed your tags, or your page was not available during a crawl, ask for a new crawl. Use the Sharing Debugger or a Graph API call with scrape=true, as that Meta page describes.

To see what the crawler gets, set the shell variable URL to your page address. Then run the command that Meta gives:

curl -v --compressed -H "Range: bytes=0-524288" -H "Connection: close" -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "$URL"

Verify requests

Meta’s crawler page does not publish an IP list. Meta’s networks use the autonomous system number AS32934 (PeeringDB). An autonomous system number identifies the networks of one operator. If you want to make sure that an address belongs to Meta, list the routes that are registered for AS32934:

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

The list is long, and it changes. Run the query again before you update firewall rules.

Other Meta crawlers

Meta runs other crawlers with other purposes, and each one has its own robots.txt token. A rule for facebookexternalhit does not apply to them. Meta-ExternalAgent crawls for AI model training and product indexing. Meta-ExternalFetcher fetches single links at the request of a user. Meta-WebIndexer crawls to improve Meta AI search results. Meta-ExternalAds crawls for advertising products. For the older FacebookBot name, see FacebookBot.