Exa AI: Search API, ExaSearchBot Crawler and Opt-Out

Exa is a web search API for AI applications. It runs its own search index, which its crawler ExaSearchBot fills. A second user agent, ExaUser, fetches a page live when a user or an app asks for it. If you want your pages in Exa results, allow ExaSearchBot. If you do not, block it in robots.txt. For pages that Exa already has, add noindex first and block the crawler after Exa drops them.

What Exa does

The Exa FAQ says that Exa gives web search and research infrastructure to AI applications. It combines an independent search index with APIs for content extraction and multi-step research. Its Search API takes natural-language queries, searches by meaning and can return page contents in the same request.

Exa returns the source URLs and the content it used, so an application can cite its sources. Apps can limit a search to some domains with includeDomains, or remove domains with excludeDomains. Exa says that in October 2026 its index tracked over 2 trillion URLs and served 100 billion pages.

Exa user agents

User agent	Purpose	robots.txt
ExaSearchBot	Finds and indexes public pages for Exa search	Follows robots.txt, token ExaSearchBot
ExaUser	Fetches a page when a user or an app asks for it	Robots.txt rules do not always apply

The ExaSearchBot page gives this user-agent string:

Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)

The ExaUser page gives this one:

Mozilla/5.0 (compatible; ExaUser/1.0; +https://crawler.exa.ai/exauser)

Exa says that ExaUser does not crawl automatically and does not decide whether a page can appear in Exa search. Because a user starts each ExaUser fetch, robots.txt rules do not always apply to it. Use the ExaSearchBot token to manage search opt-outs.

Block ExaSearchBot in robots.txt

ExaSearchBot first looks for a robots.txt group with its own name. If there is none, Exa says that it follows the rules you set for major search engines. Otherwise it follows the * group.

If you want to keep Exa out of your whole site, add this group:

User-agent: ExaSearchBot
Disallow: /

If you want to block only one folder, use a path:

User-agent: ExaSearchBot
Disallow: /private/

Exa says that ExaSearchBot limits its request rate for each site. It does not try to get past logins, paywalls or CAPTCHAs, and it submits no forms.

Remove pages already in Exa

Exa says that robots.txt controls crawling, not indexing. If you want to remove a page that Exa already has, add a noindex robots meta tag or an X-Robots-Tag: noindex header to the page. ExaSearchBot drops the page after its next fetch. So the page must stay open to ExaSearchBot until that fetch. The meta robots guide shows the tag.

Verify Exa requests

Exa signs every ExaSearchBot and ExaUser request with HTTP Message Signatures (RFC 9421), under the Web Bot Auth scheme. Each request carries Signature, Signature-Input and Signature-Agent headers. Cloudflare, Akamai and other providers that support Web Bot Auth validate these signatures automatically.

If you want to make sure that a request came from Exa, get the Ed25519 public keys from the key directory. Then make sure that the request signature is valid for one of these keys, as RFC 9421 describes. The crawler page lists no IP ranges. Exa says that the signature check works whatever the IP address of the request is.

For questions, Exa asks site owners to email [email protected] from an address at the domain in question.