Yep: Ahrefs Search Engine, Search API and AhrefsBot

Yep is a search engine from Ahrefs, a company that makes SEO tools. In 2025 Yep moved its focus to a web search API for AI agents and apps. AhrefsBot crawls the pages for both. If you want your pages in Yep, let AhrefsBot crawl them. If you do not, block AhrefsBot in robots.txt.

From search engine to search API

The timeline on the Yep Search API site says that Ahrefs announced Yep in 2019. The plan was to share ad revenue with content creators. In June 2022, TechCrunch reported that Yep would give 90% of its ad revenue to content creators.

The same timeline tells what changed. It says that LLMs, AI Overviews and ChatGPT brought “challenges to the original revenue-sharing model”. It also says that Yep moved its focus to a Search API in 2025. The Search API pages do not mention revenue sharing.

The Ahrefs crawler page still calls Yep “an independent, privacy-focused search engine”. Neither Ahrefs nor Yep says on these pages that the consumer search at yep.com has closed.

The Yep Search API

The Ahrefs page about Yep describes the API for AI agents, RAG pipelines and search apps. RAG (retrieval-augmented generation) means that an app gives search results to a language model as context. These facts about the API come from that page:

• It runs on the Ahrefs crawl index, and Ahrefs operates the crawler, the index and the ranking.
• A basic search returns the URL, meta title and meta description of each result.
• A search with the highlights type also returns passages from the page that match the query.
• Developers can filter results by language, country, content type, domain, publish date and crawl date.

So the title, the meta description and the main text of a page can all go to an AI app through Yep. Each app decides how it shows and links these sources.

AhrefsBot is the Yep crawler

The Ahrefs crawler page says that AhrefsBot “powers the database” for the Ahrefs tools and for Yep. It gives this user-agent string:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

The robots.txt token is AhrefsBot. Ahrefs says that AhrefsBot obeys robots.txt Allow and Disallow rules and the Crawl-delay rule.

The AhrefsBot guide has the IP lists, reverse DNS check and other Ahrefs bots.

Block AhrefsBot and opt out of Yep

If you want to keep your site out of Yep, add this group to robots.txt:

User-agent: AhrefsBot
Disallow: /

This rule also stops the crawl that feeds the Ahrefs SEO tools. Ahrefs does not document a separate token that blocks Yep only.

If you want AhrefsBot to crawl more slowly instead, add a Crawl-delay rule:

User-agent: AhrefsBot
Crawl-delay: 10

Tell Yep about new or changed pages

Yep takes part in IndexNow. IndexNow is a protocol that lets a site tell search engines that a URL changed. The IndexNow list of search engines gives this endpoint for Yep:

https://indexnow.yep.com/indexnow?url={your-url}&key={your-key}

To use IndexNow, put a key file on your site, then send the changed URLs with that key. The IndexNow FAQ explains the steps. It also says that a submission to one search engine is shared with all search engines that use IndexNow.