# Applebot: Apple Web Crawler, User Agent and robots.txt

> Applebot crawls for Spotlight, Siri and Safari search and Apple AI features. User agent format, DNS and IP verification, robots.txt and meta tags.

Applebot is the web crawler of Apple. Its data powers search in Spotlight, Siri and Safari, and Apple also uses it for generative AI features. If you want your pages in Apple search, allow Applebot. To limit the AI use, use the [Applebot-Extended](/ai-crawler-bot/applebot-extended/) token and the `nosnippet` tag.

## What Apple uses Applebot data for

Apple lists three uses on its [About Applebot](https://support.apple.com/en-us/119829) page:

| Use | How to opt out |
|---|---|
| Search in Spotlight, Siri and Safari | Disallow `Applebot` in robots.txt, or use `noindex` |
| Training of Apple foundation models for Apple Intelligence, Services and Developer Tools | Disallow the [Applebot-Extended](/ai-crawler-bot/applebot-extended/) token in robots.txt |
| Current context for AI answers in Siri and Search, which can link to sources | `nosnippet` meta tag on the content |

Apple states that a site can opt out of the two AI uses and still let Applebot crawl. In that case, the content stays in Spotlight, Siri and Safari.

## User agent string

The robots.txt token is `Applebot`. The search crawler uses this general format:

```
Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)
```

Desktop example from Apple:

```
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
```

Mobile example from Apple:

```
Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
```

Apple sometimes changes the browser version in the string. The general format stays the same.

Applebot also powers other Apple user agents, such as Podcasts. One special agent is `iTMS`. It crawls only URLs of content registered on Apple Podcasts, and it does not follow robots.txt.

Apple does not document a crawler named `Apple-CloudKit`. [CloudKit](https://developer.apple.com/icloud/cloudkit/) is an Apple service that stores app data in iCloud. If a request with this name fails the Applebot checks below, treat it as an unknown client.

## Verify Applebot requests

Anyone can copy a user agent string. To make sure that a request came from Apple, use one of the two methods that Apple documents:

1. Do a reverse DNS lookup on the IP address. The host name must end in `applebot.apple.com`.
2. Do a forward DNS lookup on that host name. Make sure that it returns the same IP address.

This is Apple's example:

```
$ host 17.58.101.179
179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
$ host 17-58-101-179.applebot.apple.com
17-58-101-179.applebot.apple.com has address 17.58.101.179.
```

The second method is to compare the IP address with the CIDR prefixes in Apple's [applebot.json](https://search.developer.apple.com/applebot.json) file.

## robots.txt rules for Applebot

Applebot follows robots.txt rules that name it. In this example from Apple, Applebot does not crawl `/private/` or `/not-allowed/`:

```
User-agent: Applebot
Allow: /
Disallow: /private/

User-agent: *
Disallow: /not-allowed/
```

If you want to block Applebot from the whole site, use this group:

```
User-agent: Applebot
Disallow: /
```

Apple documents these details:

- If robots.txt does not name Applebot but names Googlebot, Applebot follows the Googlebot rules.
- Applebot does not follow `Crawl-delay`. It slows down when a site becomes slow or returns errors. Apple also caches crawled content to reduce unnecessary crawling.
- Applebot can render pages in a browser. If robots.txt blocks the JavaScript, CSS or XHR files that a page needs, Applebot can fail to render it.

## Meta tags and headers

Applebot supports robots meta tags in the ` ` of a page. You can use `name="robots"` for all crawlers or `name="applebot"` for Apple only.

| Directive | Effect for Applebot |
|---|---|
| `noindex` | The page does not appear in Spotlight or Siri Suggestions. |
| `nosnippet` | No description or web answer. Apple does not use the content as context for AI answers. |
| `nofollow` | Applebot does not follow links on the page. |
| `none` | Same as `noindex`, `nosnippet` and `nofollow` together. |

This tag keeps a page out of the AI answers in Siri and Search, but leaves it in Apple search:

```html

```

For PDF files and other files without HTML, send the directive in an HTTP header, for example `X-Robots-Tag: applebot: nosnippet`.

For paywalled pages, Apple supports the schema.org property `"isAccessibleForFree": false` in JSON-LD. Apple does not use these pages as AI context, but the pages can appear in search. The signal applies to the whole page. Apple does not support markup for one section of a page with `hasPart`.

Apple's contact address for crawler questions is applebot@apple.com.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/applebot/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)