How to Find and Block AI Crawlers on Your Site
Find AI crawlers in your access log, match their IP addresses to operator lists, and control them with robots.txt and firewall rules.
5 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 6
Guides to AI crawlers and bots that index your website for AI training and search.
Kangaroo Bot: Kangaroo LLM Crawler and robots.txt
Kangaroo Bot is the crawler of Kangaroo LLM, an Australian open-source LLM project. Its robots.txt token, the project status and how to block it.
3 min read
LinkedInBot: LinkedIn Link Preview Bot User Agent
LinkedInBot reads a page when someone shares its URL on LinkedIn, to build the link preview. User agents, Open Graph tags and robots.txt.
3 min read
Meta-ExternalAgent: Meta AI Crawler and robots.txt
Meta-ExternalAgent crawls the web to train Meta AI models and index content. User agent, robots.txt rules and what a block changes.
3 min read
Meta-ExternalFetcher: User Agent and How to Block It
Meta-ExternalFetcher fetches single links when a user asks a Meta product. It can bypass robots.txt. User agent and server blocking.
3 min read
MJ12bot: Majestic Crawler User Agent and robots.txt
MJ12bot is the Majestic backlink crawler. It obeys robots.txt and Crawl-Delay up to 20 seconds. How to block, slow and identify it.
3 min read
MLBot: Undocumented Crawler User Agent and Blocking
MLBot has no operator documentation. What public bot lists say, what is unknown, and how to block it with robots.txt and nginx.
2 min read
MojeekBot: Mojeek Crawler User Agent and robots.txt
MojeekBot crawls for the Mojeek search engine. robots.txt token, noindex support, crawl rate and how to identify real requests.
2 min read
MSNBot: Retired Microsoft Crawler Replaced by Bingbot
MSNBot was renamed Bingbot on October 1, 2010. Old msnbot robots.txt rules, msnbot host names, and how to identify Bing requests.
2 min read