FriendlyCrawler: Unknown AI Dataset Crawler and Blocking
FriendlyCrawler has no known operator. Bot lists say it builds machine learning datasets. What is known and how to block it in robots.txt.
Table of Contents
FriendlyCrawler is a web crawler whose operator is not publicly known. Community bot lists say that it collects data to build machine learning datasets. If you do not want it on your site, disallow FriendlyCrawler in robots.txt and block the name on your server.
What is known
The ai.robots.txt project keeps a public list of AI crawlers and their robots.txt names. Its robots.json file has this entry for FriendlyCrawler:
| Field | Value in the list |
|---|---|
| Operator | Unknown |
| Function | Builds datasets for machine learning experiments |
| Follows robots.txt | Yes, with a January 2024 blog post by Alex Kunz as the source |
| Frequency | Unclear |
The list uses the name FriendlyCrawler as the robots.txt token. A token is the name in a robots.txt User-agent line.
What is not known
We found no documentation from the operator. We also found no published user-agent string, IP list or reverse DNS method for this crawler. For this reason, you cannot prove where a request with this name comes from. Other clients can use the same name.
Block FriendlyCrawler
If you want to keep it out of your whole site, add this group to /robots.txt:
User-agent: FriendlyCrawler
Disallow: /If you want a block that does not depend on the crawler, also return 403 to the user agent on your server. For example, in nginx, add this to the server block:
if ($http_user_agent ~* "FriendlyCrawler") {
return 403;
}These rules match only FriendlyCrawler. They do not change search engine crawlers such as Googlebot. For rules that cover many AI crawlers at once, see the robots.txt guide for AI crawlers.