Brave Search AI Answers: Crawler, Index and Opt-Out
Brave Search uses its own index for AI Answers and Ask Brave. How the Brave crawler follows Googlebot rules and how to remove a page.
Brave Search is the search engine of Brave, with its own web index. It shows AI Answers and an AI chat mode, Ask Brave, with links to the sources. The Brave crawler does not use its own user agent. It obeys the robots.txt rules for Googlebot, and it removes a page from the index when the page has a noindex directive.
AI features in Brave Search
The AI in Brave Search help page lists these features:
| Feature | What Brave says it does |
|---|---|
| AI Answers | Gives a short summary answer with references to the sources of the main claims |
| Ask Brave | Gives longer answers, follow-up chat and a Deep Research mode |
| Featured snippets | Shows a text extract from one page that answers the question |
| AI-powered descriptions | Writes descriptions of search results with a question answering model |
Brave says that it always shows where the information in AI Answers comes from. Brave also says that parts of AI Answers use open-source large language models, and that its search AI uses its own models and algorithms.
How Brave finds pages
Brave Search has its own crawler. The Brave Search crawler page says that the crawler also uses data from the Web Discovery Project. This is a setting in the Brave browser that is off by default. Users who turn it on give information to Brave. Brave says that the system keeps it from knowing who gave which information.
For some queries, Brave Search has too few results of its own. In this case, a user can let the Brave browser check Google for the same query in an anonymous way. Brave calls this Google fallback mixing.
Brave also sells access to its index through the Brave Search API. Brave says that AI labs use the API to ground answers. The API FAQ says that customers can store results or train models with a plan that includes storage rights. Brave documents no separate opt-out for this AI use.
Brave crawler and robots.txt
Brave says that its crawler does not show a separate user agent. The reason is that some sites allow only Google to crawl them. Brave also says that its crawler does not crawl a domain or page that Googlebot cannot crawl.
As a result, your rules for Googlebot also apply to Brave. For example, this group keeps both Googlebot and the Brave crawler out of /private/:
User-agent: Googlebot
Disallow: /private/The Brave crawler page names no robots.txt token that applies only to Brave.
The name Bravebot
Some bot lists use the name Bravebot for the Brave crawler. Cloudflare Radar lists Bravebot with this user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36Brave’s crawler page does not mention this string or a Bravebot token. Brave also publishes no IP list or DNS verification method on that page. A User-agent: Bravebot group in robots.txt has no documented effect.
Remove a page from Brave Search
Brave says that robots.txt does not stop a page from being indexed. To remove a page from Brave Search, add a noindex directive:
<meta name="robots" content="noindex">Brave must fetch the page again before it removes the page. Brave has a form on its crawler page where you can ask for a new fetch. Do not block the page in robots.txt at the same time, because then the crawler cannot read the tag. The meta robots guide explains the tag.
If Brave Search shows a page that no longer exists, use the form on the crawler page or write to [email protected].