360Spider: Qihoo 360 Crawler User Agent and Blocking
360Spider crawls the web for 360 Search (so.com). User agents, robots.txt tokens, the Indexpage extension, published IP ranges and blocking.
3 min read
AI CRAWLER BOTS
AI Crawler Bots
Guides to AI crawlers and bots that index your website for AI training and search.
Adbeat Crawler (adbeat_bot): User Agent and Blocking
adbeat_bot collects ad data for Adbeat, a display ad intelligence service. Its robots.txt token, user agent, and how to block it.
2 min read
AdIdxBot: Bing Ads Crawler User Agent and robots.txt
AdIdxBot is the Bing Ads crawler that checks ads and the pages they link to. User agent strings, robots.txt rules and how to verify requests.
3 min read
AhrefsBot: Ahrefs Crawler User Agent and robots.txt
AhrefsBot crawls for the Ahrefs SEO tools and the Yep search engine. User agent, IP lists, reverse DNS and robots.txt rules to block or slow it.
3 min read
AI2Bot: Ai2 Crawler User Agent and robots.txt
AI2Bot is the crawler of Ai2 (Allen Institute for AI). It collects content to train open models such as Olmo. User agent and how to block it.
3 min read
Amazonbot: Amazon Crawler User Agent and robots.txt
Amazonbot, Amzn-SearchBot and Amzn-User: what each Amazon agent does, user agent strings, IP lists, and how to block them in robots.txt.
3 min read
Applebot-Extended: Opt Out of Apple AI Training
Applebot-Extended is a robots.txt token, not a crawler. Disallow it to keep Apple from training its AI models on your pages. Apple search stays the same.
4 min read
Applebot: Apple Web Crawler, User Agent and robots.txt
Applebot crawls for Spotlight, Siri and Safari search and Apple AI features. User agent format, DNS and IP verification, robots.txt and meta tags.
4 min read
archive.org_bot: Internet Archive Crawler and robots.txt
archive.org_bot crawls pages for the Wayback Machine. Its user agent token, the limits of robots.txt, and how to request removal of archived pages.
3 min read