Proximic: Comscore Ad Crawler User Agent and robots.txt
The proximic crawler analyzes pages for Comscore contextual advertising. User agent, robots.txt rules, crawl delay and what a block changes.
3 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 8
Guides to AI crawlers and bots that index your website for AI training and search.
Rogerbot: Moz Site Audit Crawler and robots.txt
Rogerbot is the Moz Pro site audit crawler, separate from the Dotbot link crawler. How to allow, block or slow it in robots.txt and how to identify it.
3 min read
Screaming Frog SEO Spider: User Agent and robots.txt
Screaming Frog SEO Spider is a desktop crawler that people run on their own computers. Its user agent, robots.txt handling and how to block it.
3 min read
SecurityTrails Bot: What Is Known and How to Block It
SecurityTrails sells DNS, WHOIS and IP data to security teams. What is documented about a SecurityTrails bot and how to handle its requests.
2 min read
SemrushBot: Semrush Crawler User Agents and robots.txt
SemrushBot and the other Semrush bots collect data for Semrush SEO tools. Their robots.txt tokens, Crawl-delay limit and how to block each one.
3 min read
SeobilityBot: Seobility SEO Crawler and robots.txt
SeobilityBot crawls sites for the Seobility SEO tools. Its user agent, IP list and robots.txt rules to block it or to let only it crawl a site.
2 min read
Serpstatbot: Serpstat Backlink Crawler and robots.txt
Serpstatbot crawls the web for the Serpstat backlink index. Its robots.txt token, Crawl-delay limit, IP list and why a robots.txt block can fail.
3 min read
SeznamBot: Seznam.cz Crawler User Agent and robots.txt
SeznamBot crawls for Seznam.cz search. Its user agent, IP list, reverse DNS names and robots.txt rules, including the Request-rate directive.
4 min read
SISTRIX Crawler: User Agent, Reverse DNS and robots.txt
The SISTRIX Crawler fetches pages for the SISTRIX SEO Toolbox. Its default user agent, reverse DNS names, robots.txt rules and project settings.
3 min read