Serpstatbot: Serpstat Backlink Crawler and robots.txt

Serpstatbot is the crawler of Serpstat, an SEO platform. It crawls the web to find new links and track changes in the Serpstat backlink database. Serpstat says that serpstatbot follows robots.txt. Serpstat also publishes a list of its IP addresses.

What serpstatbot does

The serpstatbot FAQ gives these facts:

• The crawled data goes into the Serpstat backlink index. Serpstat users can read it in Serpstat.
• The bot keeps following links to pages that return 404 or 301, as long as the links still exist.
• The bot also crawls links with the rel="nofollow" attribute.

User agent

The FAQ uses the robots.txt token serpstatbot but does not give the full user agent string. The community list crawler-user-agents matches the bot by the text serpstatbot/ in the user agent.

Block serpstatbot in robots.txt

If you want to block serpstatbot, add this group:

User-agent: serpstatbot
Disallow: /

Make sure that serpstatbot can download your robots.txt file. If the download fails, for example with 403 Forbidden, serpstatbot assumes that there are no restrictions. It then crawls all pages that it can reach.

serpstatbot also supports these rules:

• Redirects of robots.txt within the same site. It does not follow a redirect to another domain.
• Simple wildcard patterns in Disallow lines.
• An Allow line wins over a Disallow line if the Allow line is longer.

Slow serpstatbot down

If you want to slow serpstatbot down, set a delay in whole seconds between two requests:

User-agent: serpstatbot
Crawl-Delay: 5

Serpstat accepts a delay of up to 20 seconds. It cuts higher values to 20. It also follows a Crawl-Delay in the group for all bots (*). If it finds a Crawl-Delay for any other bot, it slows down by itself.

When a block seems to fail

Serpstat gives two common causes:

• robots.txt redirects to a different domain. serpstatbot follows only redirects on the same domain.
• Several domains on one server write to one access log without the domain name. Requests for another domain then look like requests to your site. Add the domain name to the log format, or keep one log for each domain.

IP addresses

Serpstat publishes a list of serpstatbot IP addresses. Use it to make sure that a request comes from Serpstat, or use it in firewall rules.

If serpstatbot ignores your robots.txt, Serpstat asks you to email [email protected]. Send the URL of your site and the log entries.