# FriendlyCrawler: Unknown AI Dataset Crawler and Blocking

> FriendlyCrawler has no known operator. Bot lists say it builds machine learning datasets. What is known and how to block it in robots.txt.

FriendlyCrawler is a web crawler whose operator is not publicly known. Community bot lists say that it collects data to build machine learning datasets. If you do not want it on your site, disallow `FriendlyCrawler` in robots.txt and block the name on your server.

## What is known

The [ai.robots.txt project](https://github.com/ai-robots-txt/ai.robots.txt) keeps a public list of AI crawlers and their robots.txt names. Its [robots.json](https://github.com/ai-robots-txt/ai.robots.txt/blob/main/robots.json) file has this entry for FriendlyCrawler:

| Field | Value in the list |
|---|---|
| Operator | Unknown |
| Function | Builds datasets for machine learning experiments |
| Follows robots.txt | Yes, with a [January 2024 blog post by Alex Kunz](https://imho.alex-kunz.com/2024/01/25/an-update-on-friendly-crawler/) as the source |
| Frequency | Unclear |

The list uses the name `FriendlyCrawler` as the robots.txt token. A token is the name in a robots.txt `User-agent` line.

## What is not known

We found no documentation from the operator. We also found no published user-agent string, IP list or reverse DNS method for this crawler. For this reason, you cannot prove where a request with this name comes from. Other clients can use the same name.

## Block FriendlyCrawler

If you want to keep it out of your whole site, add this group to `/robots.txt`:

```text
User-agent: FriendlyCrawler
Disallow: /
```

If you want a block that does not depend on the crawler, also return `403` to the user agent on your server. For example, in nginx, add this to the `server` block:

```nginx
if ($http_user_agent ~* "FriendlyCrawler") {
    return 403;
}
```

These rules match only FriendlyCrawler. They do not change search engine crawlers such as [Googlebot](/ai-crawler-bot/googlebot/). For rules that cover many AI crawlers at once, see the [robots.txt guide for AI crawlers](/guide/robots-txt-ai-crawlers/).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/friendlycrawler/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)