Applebot: Apple Web Crawler, User Agent and robots.txt

Applebot is the web crawler of Apple. Its data powers search in Spotlight, Siri and Safari, and Apple also uses it for generative AI features. If you want your pages in Apple search, allow Applebot. To limit the AI use, use the Applebot-Extended token and the nosnippet tag.

What Apple uses Applebot data for

Apple lists three uses on its About Applebot page:

Use	How to opt out
Search in Spotlight, Siri and Safari	Disallow Applebot in robots.txt, or use noindex
Training of Apple foundation models for Apple Intelligence, Services and Developer Tools	Disallow the Applebot-Extended token in robots.txt
Current context for AI answers in Siri and Search, which can link to sources	nosnippet meta tag on the content

Apple states that a site can opt out of the two AI uses and still let Applebot crawl. In that case, the content stays in Spotlight, Siri and Safari.

User agent string

The robots.txt token is Applebot. The search crawler uses this general format:

Mozilla/5.0 (Device; OS_version) AppleWebKit/WebKit_version (KHTML, like Gecko)Version/Safari_version [Mobile/Mobile_version] Safari/WebKit_version (Applebot/Applebot_version; +http://www.apple.com/go/applebot)

Desktop example from Apple:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Mobile example from Apple:

Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)

Apple sometimes changes the browser version in the string. The general format stays the same.

Applebot also powers other Apple user agents, such as Podcasts. One special agent is iTMS. It crawls only URLs of content registered on Apple Podcasts, and it does not follow robots.txt.

Apple does not document a crawler named Apple-CloudKit. CloudKit is an Apple service that stores app data in iCloud. If a request with this name fails the Applebot checks below, treat it as an unknown client.

Verify Applebot requests

Anyone can copy a user agent string. To make sure that a request came from Apple, use one of the two methods that Apple documents:

1. Do a reverse DNS lookup on the IP address. The host name must end in applebot.apple.com.
2. Do a forward DNS lookup on that host name. Make sure that it returns the same IP address.

This is Apple’s example:

$ host 17.58.101.179
179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
$ host 17-58-101-179.applebot.apple.com
17-58-101-179.applebot.apple.com has address 17.58.101.179.

The second method is to compare the IP address with the CIDR prefixes in Apple’s applebot.json file.

robots.txt rules for Applebot

Applebot follows robots.txt rules that name it. In this example from Apple, Applebot does not crawl /private/ or /not-allowed/:

User-agent: Applebot
Allow: /
Disallow: /private/

User-agent: *
Disallow: /not-allowed/

If you want to block Applebot from the whole site, use this group:

User-agent: Applebot
Disallow: /

Apple documents these details:

• If robots.txt does not name Applebot but names Googlebot, Applebot follows the Googlebot rules.
• Applebot does not follow Crawl-delay. It slows down when a site becomes slow or returns errors. Apple also caches crawled content to reduce unnecessary crawling.
• Applebot can render pages in a browser. If robots.txt blocks the JavaScript, CSS or XHR files that a page needs, Applebot can fail to render it.

Meta tags and headers

Applebot supports robots meta tags in the <head> of a page. You can use name="robots" for all crawlers or name="applebot" for Apple only.

Directive	Effect for Applebot
noindex	The page does not appear in Spotlight or Siri Suggestions.
nosnippet	No description or web answer. Apple does not use the content as context for AI answers.
nofollow	Applebot does not follow links on the page.
none	Same as noindex, nosnippet and nofollow together.

This tag keeps a page out of the AI answers in Siri and Search, but leaves it in Apple search:

<meta name="applebot" content="nosnippet">

For PDF files and other files without HTML, send the directive in an HTTP header, for example X-Robots-Tag: applebot: nosnippet.

For paywalled pages, Apple supports the schema.org property "isAccessibleForFree": false in JSON-LD. Apple does not use these pages as AI context, but the pages can appear in search. The signal applies to the whole page. Apple does not support markup for one section of a page with hasPart.

Apple’s contact address for crawler questions is [email protected].