AhrefsBot: Ahrefs Crawler User Agent and robots.txt

AhrefsBot is the web crawler of Ahrefs, a company that makes SEO and marketing tools. Its data goes into the Ahrefs database and into Yep, the search engine of Ahrefs. Ahrefs says that AhrefsBot follows robots.txt, so you can slow it down or block it there.

Ahrefs bots

Ahrefs lists two crawlers on its bot page:

Bot	robots.txt token	What it is for
AhrefsBot	AhrefsBot	Builds the database for the Ahrefs tools and for the Yep search engine
AhrefsSiteAudit	AhrefsSiteAudit	Runs the Ahrefs Site Audit tool, which finds technical and on-page SEO problems

The user agent strings are:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

AhrefsSiteAudit also has a mobile user agent. It contains the same AhrefsSiteAudit/6.1 part.

How AhrefsBot reads robots.txt

Ahrefs says that both bots follow Allow, Disallow and Crawl-delay. It also gives these limits:

• AhrefsBot follows Crawl-delay for HTML pages. When it renders a page, it can request images, scripts and style sheets at the same time.
• AhrefsBot needs some time to read a changed robots.txt. It reads the file before its next scheduled crawl.
• If your robots.txt has errors, the bots cannot read your rules and continue to crawl as before.
• A site owner who verified the site in Ahrefs can let AhrefsSiteAudit ignore robots.txt on that site.

Block or slow AhrefsBot

If you want to block AhrefsBot from the whole site, add this group:

User-agent: AhrefsBot
Disallow: /

If you want to slow it down, set the minimum delay between two requests in seconds:

User-agent: AhrefsBot
Crawl-delay: 10

If your server has an outage, you can return 4xx or 5xx status codes for a short time. Ahrefs says that AhrefsBot then crawls the site more slowly.

AhrefsBot also feeds the Yep search engine. If you block AhrefsBot, you also stop the crawl that Yep gets its data from. Ahrefs lists a separate robots.txt token for each bot. Write a group for each bot that you want to control.

Verify AhrefsBot requests

Ahrefs crawls from published IP addresses. It gives two lists:

• IP ranges: api.ahrefs.com/v3/public/crawler-ip-ranges
• Single IP addresses: api.ahrefs.com/v3/public/crawler-ips

Ahrefs says that the reverse DNS name of its crawler IP addresses always ends in ahrefs.com or ahrefs.net. If a request has the AhrefsBot user agent, make sure that one of these conditions is true:

• The IP address is in one of the two lists.
• The reverse DNS name of the IP address ends in ahrefs.com or ahrefs.net.

Ahrefs also says that Cloudflare lists both bots as verified bots.