SISTRIX Crawler: User Agent, Reverse DNS and robots.txt
The SISTRIX Crawler fetches pages for the SISTRIX SEO Toolbox. Its default user agent, reverse DNS names, robots.txt rules and project settings.
The SISTRIX Crawler fetches pages for the SISTRIX Toolbox, a set of SEO tools from SISTRIX. It reads robots.txt and follows rules for sistrix and for all bots. Its IP addresses have reverse DNS names in the domain sistrix.net.
User agent
The SISTRIX tutorial Crawling Errors in the Onpage Projects says that the SISTRIX Crawler does all access for the SISTRIX Toolbox. The default user agent is:
Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)SISTRIX users can change the user agent and the crawl speed for their own Onpage projects, under “Project-Management > Crawler”. So a crawl for a SISTRIX project can arrive with a different user agent.
Identify real SISTRIX requests
SISTRIX says that all IP addresses of its crawler have a reverse DNS name in the domain sistrix.net. Its example: the IP address 136.243.92.8 returns 136-243-92-8.crawler.sistrix.net.
- Run a reverse DNS lookup (a query for the hostname of an IP address) on the request IP.
- Make sure that the hostname ends in
.sistrix.net. - Run a forward DNS lookup on that hostname.
- Make sure that it returns the same IP address.
A SISTRIX user can see the crawler IP for a project in the crawler settings of that project.
robots.txt rules
Before its first request to a site, the crawler downloads robots.txt from the root of each hostname. It follows rules for sistrix and rules for all bots (*). If robots.txt names a sitemap, the crawler uses the sitemap as a starting point.
If you want to block the SISTRIX Crawler, add this group:
User-agent: sistrix
Disallow: /If a SISTRIX user changed the user agent of a project crawler, rules for that user agent also apply to the crawl.
Other crawler behavior
SISTRIX describes these points in the same tutorial:
- The crawler measures how fast pages load and adjusts its request rate, so it does not overload the server.
- It does not save cookies.
- It does not run JavaScript. It reads the static HTML.
- All SISTRIX crawlers are in Germany. Geo-IP rules can show them German content or block them.
Block at the server
You can also block the crawler on the server, for example in the .htaccess file of an Apache server. The crawler then gets HTTP status 403. SISTRIX says that after a 403 on the first request, the crawler cannot access any pages on that server.