PerplexityBot: Perplexity Search Crawler and robots.txt

PerplexityBot is the crawler that Perplexity uses to find and link websites in its search results. Perplexity says that it does not use PerplexityBot to crawl content for AI foundation models. If you want your pages to appear in Perplexity answers, allow PerplexityBot in robots.txt and in your firewall.

What PerplexityBot does

The crawler documentation says that PerplexityBot finds and links websites for Perplexity search results. It names PerplexityBot as a robots.txt token, and it says that each robots.txt setting works independently. The Perplexity AI guide covers the search product.

Some third-party lists also show the name Perplexity-Ads-Bot. Perplexity does not document an ads crawler. If a request uses that name, treat it as unverified unless its IP address is in a Perplexity IP list.

User-agent string

A user agent is the client name in the User-Agent HTTP header. Perplexity gives this full string for PerplexityBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

The robots.txt token is PerplexityBot.

Block or allow PerplexityBot in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. If you want to block PerplexityBot on the whole site, add this group:

User-agent: PerplexityBot
Disallow: /

If you want to allow it, add this group:

User-agent: PerplexityBot
Allow: /

Perplexity says that its systems can take up to 24 hours to show a robots.txt change.

What a block changes

Perplexity recommends that you allow PerplexityBot if you want your site to appear in its search results. A block works against that goal.

A PerplexityBot group does not apply to Perplexity-User, the separate agent that opens a page when a user asks about it.

Verify requests

Anyone can copy a user-agent string. Perplexity publishes IP addresses, not reverse DNS names. The PerplexityBot addresses are at perplexity.com/perplexitybot.json. If a request has PerplexityBot in the user agent, make sure that its IP address is in this list.

Firewall rules

A web application firewall (WAF) filters HTTP requests before they reach your site. Perplexity says that a WAF can block its crawler, and it gives steps for Cloudflare and AWS WAF. Its advice is to match the user agent and the published IP ranges together, then set the action to Allow. In AWS WAF, the Allow rule must have a higher priority than your blocking rules.

The IP list changes. The documentation recommends an automated job that fetches the list and updates your WAF rules.

If you want to block PerplexityBot at the firewall instead, use the same two conditions with a block action. For rules that cover many AI crawlers, see robots.txt for AI crawlers.