SemrushBot: Semrush Crawler User Agents and robots.txt
SemrushBot and the other Semrush bots collect data for Semrush SEO tools. Their robots.txt tokens, Crawl-delay limit and how to block each one.
SemrushBot is the crawler of Semrush, a company that sells SEO and marketing tools. Its data goes into the Semrush backlink index and several Semrush tools. Semrush documents a separate robots.txt token for each of its bots, so you can block them one at a time.
Semrush bots and their tokens
The Semrush bot page lists these bots:
| robots.txt token | What it collects data for |
|---|---|
SemrushBot |
Backlink Analytics (the Semrush index of links) and other tools |
SiteAuditBot |
Site Audit, which finds SEO, technical and usability issues |
SemrushBot-BA |
Backlink Audit |
SemrushBot-SI |
On Page SEO Checker and similar tools |
SemrushBot-SWA |
URL access tests for the SEO Writing Assistant |
SplitSignalBot |
SEO A/B tests in SplitSignal |
SemrushBot-OCOB |
Content Toolkit |
SemrushBot-FT |
Plagiarism Checker and similar tools |
RyteBot |
Tools provided by Ryte.com |
SemrushBot-ESI |
Semrush Enterprise Site Intelligence |
Semrush also lists Link Building, Topic Research and the SEO Content Template among the tools that use SemrushBot data.
User agent string
The Semrush bot page lists robots.txt tokens, but not full user agent strings. The community list crawler-user-agents records strings like this one:
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)The same list shows other version numbers in this place. Match the token, not the full string.
Block SemrushBot in robots.txt
If you want to block SemrushBot for the backlink index, add this group:
User-agent: SemrushBot
Disallow: /Semrush lists a separate group for each bot. If you want to block a tool, add a group for its token. For example, this group blocks Site Audit:
User-agent: SiteAuditBot
Disallow: /SemrushBot for Backlink Analytics also supports two rules that Semrush calls non-standard extensions:
Crawl-delayup to 10 seconds. Semrush cuts higher values to 10. Without a delay, SemrushBot changes its request rate to match the load on your server.- Wildcards (
*) in paths.
Semrush gives these rules for the robots.txt file:
- Put robots.txt in the top directory of each host. Each subdomain needs its own file.
- The file must return HTTP status 200. If it returns 4xx, SemrushBot assumes that there are no restrictions. If it returns 5xx, SemrushBot does not crawl the site.
- SemrushBot can need up to one hour or 100 requests to find a change.
- If you move a site, for example from HTTP to HTTPS, move robots.txt too.
IP addresses and server rules
The Semrush bot page gives no IP list. Semrush says not to block SemrushBot by IP, because it does not use consecutive IP blocks. If you want to block it at the server or firewall, match the token in the user agent. Any client can send that name, so a match does not prove that a request came from Semrush.
Forms and old URLs
Semrush says that SemrushBot can submit login or survey forms that use the GET method. The form data is part of the URL, so the crawler can reach it. Semrush recommends the POST method for such forms.
SemrushBot can also request pages that no longer exist, if other sites still link to them.
If you think that SemrushBot ignores your robots.txt, Semrush asks you to email [email protected]. Send your URL and the log entries.