Screaming Frog SEO Spider: User Agent and robots.txt
Screaming Frog SEO Spider is a desktop crawler that people run on their own computers. Its user agent, robots.txt handling and how to block it.
Screaming Frog SEO Spider is a desktop program that crawls websites for SEO audits. A person installs it and runs it on a computer. It is not a public crawler like Googlebot, and it does not build a search index. If it visits your site, someone started a crawl of your site with it.
Who sends the requests
Each user runs the SEO Spider from their own computer or server. The requests come from the IP address of that user, not from a Screaming Frog network. So there is no IP list or reverse DNS name that you can use to identify it.
The free version crawls up to 500 URLs in one crawl, and its configuration is locked. In the licensed version, the user can change the configuration, for example the user agent and robots.txt settings below.
User agent
By default, the SEO Spider sends its own user agent, which contains the name Screaming Frog SEO Spider.
The user can change it. The configuration guide lists presets for Googlebot, Bingbot and web browsers, and a custom value. So a Screaming Frog crawl can look like Googlebot or a normal browser in your logs.
If a request claims to be Googlebot, make sure that it comes from Google. The Googlebot guide shows how to do a reverse DNS lookup for Google.
How it reads robots.txt
Screaming Frog says that by default the tool obeys robots.txt in the same way as Google. It looks for a group in this order:
- A group for
Screaming Frog SEO Spider - If there is none, a group for
Googlebot - If there is none, the group for all bots (
*)
The configuration has a separate Robots User-Agent setting. This name selects the robots.txt rules, and it can differ from the user agent in the HTTP request.
In the licensed version, the user can tell the tool to ignore robots.txt. The tool then does not download the file. So robots.txt is a request to the person who runs the crawl. It does not stop the crawl.
Block Screaming Frog
If you want to ask the tool not to crawl your site, add this group:
User-agent: Screaming Frog SEO Spider
Disallow: /The tool reads this group by default. A user can still ignore robots.txt or change the robots user agent.
If you want to stop the requests, block them at the server or firewall. A rule for the user agent Screaming Frog SEO Spider stops only crawls that keep the default name.
Crawl your own blocked site
If your site blocks all bots, you can still audit it with the free version. Add an Allow group for the tool, as Screaming Frog suggests:
User-agent: *
Disallow: /
User-agent: Screaming Frog SEO Spider
Allow: /