# YouBot: You.com Crawler User Agent and robots.txt

> YouBot crawls pages for the You.com search engine. User-agent string, robots.txt rules, IP range, reverse DNS and Web Bot Auth checks.

YouBot is the web crawler of You.com. It finds and indexes pages for the You.com search engine, and You.com says that it follows robots.txt. If you want your pages in You.com results, allow `YouBot`. If you do not, add a `Disallow` rule for `YouBot`.

## What YouBot does

The [YouBot documentation](https://docs.you.com/youbot) says that YouBot is the web crawler that powers the You.com search engine. It crawls public pages that robots.txt does not block. You.com lists these content types:

- HTML pages
- PDF documents
- Plain text files
- XML and RSS feeds
- Structured data (JSON-LD, microdata, RDFa)

The documentation does not say whether You.com uses crawled pages to train AI models. The [You.com guide](/ai-search-engine/you-com/) describes the search product.

## User-agent string

You.com gives this user-agent string. `X.X.X.X` is the version number of Chrome:

```
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36
```

The robots.txt token is `YouBot`.

## Block or limit YouBot in robots.txt

To block YouBot from the full site:

```
User-agent: YouBot
Disallow: /
```

To block YouBot from some folders:

```
User-agent: YouBot
Disallow: /private/
Disallow: /admin/
```

To slow YouBot down, You.com shows a `Crawl-delay` rule. You.com says that YouBot obeys it:

```
User-agent: YouBot
Crawl-delay: 10
```

YouBot keeps a copy of robots.txt for 30 minutes. A change to the file can take up to 30 minutes to start.

You.com does not say whether a new `Disallow` rule removes pages that YouBot already indexed. The table below shows that a 404 response removes a page from the index.

## How YouBot reacts to status codes

| Status code | What YouBot does, as You.com describes it |
|---|---|
| 200 | Crawls and indexes the page |
| 301, 302 | Follows the redirect |
| 404 | Removes the page from the index |
| 429 | Reduces the crawl rate |
| 503 | Stops crawling for a short time |

If YouBot crawls your site too much, or if you want a page crawled again, write to spider@you.com. You.com gives this address for crawl questions.

## Verify YouBot requests

Other bots can copy the YouBot user agent. You.com gives three ways to make sure that a request is real.

### Web Bot Auth signatures

YouBot signs its requests with HTTP Message Signatures. This is the Web Bot Auth standard from Cloudflare. You.com recommends this method.

1. Read the signature headers of the request.
2. Get the public keys from `https://you.com/.well-known/http-message-signatures-directory`.
3. Make sure that the signature is valid for one of these keys.

The keys are in JSON Web Key (JWK) format and use the Ed25519 algorithm.

### Reverse DNS

YouBot host names have the form `youbot-{octets-with-hyphens}.search.you.com`. For example, the IP address `68.67.112.106` has the host name `youbot-68-67-112-106.search.you.com`.

1. Do a reverse DNS lookup of the IP address in your log.
2. Make sure that the host name has the form above.
3. Do a forward lookup of that host name.
4. Make sure that the result is the same IP address.

```
host youbot-68-67-112-106.search.you.com
```

### IP range

You.com says that real YouBot requests come from `68.67.112.0/24`. If you block or allow YouBot on a firewall or CDN, use this range together with the user agent.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/youbot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)