AI CRAWLER BOTS

AI Crawler Bots - Page 4

Guides to AI crawlers and bots that index your website for AI training and search.

DotBot: Moz Link Index Crawler and robots.txt Rules

DotBot is the crawler that builds the Moz Link Index. Its robots.txt token, how to block or slow it, and how it differs from Rogerbot.

DuckAssistBot fetches pages in real time for DuckDuckGo Search Assist answers. Its user agent, IP list, and the robots.txt opt-out.

FacebookBot is not on Meta's current crawler list. What Meta documents now, which user agents to use in robots.txt, and how to block Meta crawlers.

facebookexternalhit fetches shared links for previews on Facebook, Instagram and Messenger. Its user agents, robots.txt behavior and requirements.

FeedFetcher-Google fetches RSS and Atom feeds for Google News and WebSub. It ignores robots.txt. User agent, IP list and how to block it.

Feedly Fetcher, also called FeedlyBot, gets RSS feeds for Feedly users as Feedly/1.0. It ignores robots.txt. How to block it on the server.

FriendlyCrawler has no known operator. Bot lists say it builds machine learning datasets. What is known and how to block it in robots.txt.

Google-CloudVertexBot crawls sites for Vertex AI search (Agent Search) data stores. User agent, robots.txt tokens, sitemaps and verification.

Google-Extended is a robots.txt token, not a crawler. It controls Gemini training and grounding, not Google Search ranking or AI Overviews.