SecurityTrails Bot: What Is Known and How to Block It

SecurityTrails is a data service for security teams. It sells IP, DNS, WHOIS and company data through a web platform and an API. We found no official SecurityTrails page that documents a web crawler, a user agent string, a robots.txt token or an IP list.

What SecurityTrails is

The SecurityTrails documentation calls it “a Recorded Future Company”. TechCrunch reported on January 4, 2022 that Recorded Future bought SecurityTrails. TechCrunch describes it as a company that collects current and historical domain and IP address data.

The SecurityTrails API gives access to IP, DNS, WHOIS and company data. This data is about domains, DNS records and IP addresses. It is not the text of your pages.

What is not documented

Search engines and SEO tools publish a bot page for site owners. We found no such page from SecurityTrails. So these facts are not documented:

• A user agent string
• A robots.txt token, and whether a SecurityTrails bot reads robots.txt
• An IP list or a reverse DNS name to identify its requests

Any client can send any user agent. A log line that names SecurityTrails does not prove that SecurityTrails sent it.

Identify the requests

If you see requests that name SecurityTrails, do these steps:

1. Find the IP address of the request in your server log.
2. Run a reverse DNS lookup and a WHOIS query on the IP address.
3. Make sure that the hostname or the network owner belongs to SecurityTrails or Recorded Future.

If the owner is a different company, the user agent is fake or borrowed.

Block the requests

A robots.txt rule only works if the bot reads robots.txt. SecurityTrails does not document that it does. If you want to block the requests, add a server or firewall rule for the IP address or user agent.

What blocking does not change

DNS servers answer DNS queries, not your web server. A block on your web server does not hide your DNS records. Anyone can still look up your public DNS records and the WHOIS data of your domain.