Twitterbot: X Link Preview Crawler User Agent

Twitterbot is the crawler that X, formerly Twitter, uses to fetch shared links and build card previews. It is not an AI crawler. X no longer publishes its Cards documentation, so the current behavior of the crawler is not documented. If you want link previews on X, let Twitterbot fetch your pages and preview images.

Documentation status

X once documented Cards and Twitterbot on its developer site. In October 2026, the old Cards URLs on developer.x.com redirect to the general X developer overview. The current X for Websites section covers embedded posts, buttons and widgets, but not Cards. So X does not currently document the user agent, the robots.txt behavior or the cache time of Twitterbot.

User-agent string

A user agent is the client name in the User-Agent HTTP header. Public bot lists, such as crawler-user-agents, record these strings:

Twitterbot/1.0
Twitterbot/0.1

If you match the name in logs or server rules, match Twitterbot and ignore case.

Card tags

A card is the preview box that X shows for a link in a post. Card markup uses twitter: meta tags in the page head, such as twitter:card. The Twitter Card meta tags and Open Graph meta tags guides explain the tags. Other services read the same tags. For example, the Snapchat preview service uses X/Twitter Card tags when Open Graph tags are missing.

Block Twitterbot

robots.txt is a text file at the root of your site with rules for crawlers. If you want to keep Twitterbot out of one folder, add this group to /robots.txt:

User-agent: Twitterbot
Disallow: /private/

If you want to block it from the whole site, use Disallow: / instead. X does not currently document whether Twitterbot obeys these rules. If you want to make sure that the requests stop, also block the user agent on your web server. This nginx example returns HTTP 403 for any user agent that contains Twitterbot:

if ($http_user_agent ~* "Twitterbot") {
    return 403;
}

Put the example in the server block of your site configuration, then reload nginx. Without access to your page, X cannot build a card from your tags.

Verify requests

X publishes no IP list and no reverse DNS method for Twitterbot. Anyone can copy a user-agent string. A match in your logs does not prove that the request came from X.