YouBot: You.com Crawler User Agent and robots.txt
YouBot crawls pages for the You.com search engine. User-agent string, robots.txt rules, IP range, reverse DNS and Web Bot Auth checks.
YouBot is the web crawler of You.com. It finds and indexes pages for the You.com search engine, and You.com says that it follows robots.txt. If you want your pages in You.com results, allow YouBot. If you do not, add a Disallow rule for YouBot.
What YouBot does
The YouBot documentation says that YouBot is the web crawler that powers the You.com search engine. It crawls public pages that robots.txt does not block. You.com lists these content types:
- HTML pages
- PDF documents
- Plain text files
- XML and RSS feeds
- Structured data (JSON-LD, microdata, RDFa)
The documentation does not say whether You.com uses crawled pages to train AI models. The You.com guide describes the search product.
User-agent string
You.com gives this user-agent string. X.X.X.X is the version number of Chrome:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36The robots.txt token is YouBot.
Block or limit YouBot in robots.txt
To block YouBot from the full site:
User-agent: YouBot
Disallow: /To block YouBot from some folders:
User-agent: YouBot
Disallow: /private/
Disallow: /admin/To slow YouBot down, You.com shows a Crawl-delay rule. You.com says that YouBot obeys it:
User-agent: YouBot
Crawl-delay: 10YouBot keeps a copy of robots.txt for 30 minutes. A change to the file can take up to 30 minutes to start.
You.com does not say whether a new Disallow rule removes pages that YouBot already indexed. The table below shows that a 404 response removes a page from the index.
How YouBot reacts to status codes
| Status code | What YouBot does, as You.com describes it |
|---|---|
| 200 | Crawls and indexes the page |
| 301, 302 | Follows the redirect |
| 404 | Removes the page from the index |
| 429 | Reduces the crawl rate |
| 503 | Stops crawling for a short time |
If YouBot crawls your site too much, or if you want a page crawled again, write to [email protected]. You.com gives this address for crawl questions.
Verify YouBot requests
Other bots can copy the YouBot user agent. You.com gives three ways to make sure that a request is real.
Web Bot Auth signatures
YouBot signs its requests with HTTP Message Signatures. This is the Web Bot Auth standard from Cloudflare. You.com recommends this method.
- Read the signature headers of the request.
- Get the public keys from
https://you.com/.well-known/http-message-signatures-directory. - Make sure that the signature is valid for one of these keys.
The keys are in JSON Web Key (JWK) format and use the Ed25519 algorithm.
Reverse DNS
YouBot host names have the form youbot-{octets-with-hyphens}.search.you.com. For example, the IP address 68.67.112.106 has the host name youbot-68-67-112-106.search.you.com.
- Do a reverse DNS lookup of the IP address in your log.
- Make sure that the host name has the form above.
- Do a forward lookup of that host name.
- Make sure that the result is the same IP address.
host youbot-68-67-112-106.search.you.comIP range
You.com says that real YouBot requests come from 68.67.112.0/24. If you block or allow YouBot on a firewall or CDN, use this range together with the user agent.