Claude-User: Anthropic Agent for User Requests
Claude-User opens a web page when a Claude user asks a question. Its robots.txt token, what a block changes and how to verify requests.
3 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 3
Guides to AI crawlers and bots that index your website for AI training and search.
ClaudeBot: Anthropic Crawler, User Agent and robots.txt
ClaudeBot is Anthropic's crawler for AI model training. Its robots.txt token, Anthropic's other bots, the IP list and how to block it.
3 min read
Cloudflare Always Online Crawler: User Agents and IPs
Cloudflare Always Online shows archived pages when your origin is down. The user agents and IP ranges that crawl your site, and how to turn it off.
3 min read
coccocbot: Coc Coc Crawler User Agents and robots.txt
coccocbot is the crawler family of the Coc Coc search engine. Its user agent strings, robots.txt rules, Crawl-delay limit and reverse DNS check.
3 min read
cohere-ai and cohere-training-data-crawler User Agents
cohere-ai and cohere-training-data-crawler are tokens that bot lists link to Cohere. Cohere documents neither. What Cohere says and how to block them.
3 min read
Datadog Synthetics Bot: User Agent, Headers and IPs
How to identify Datadog Synthetic Monitoring requests: the Datadog/Synthetics user agent, the sec-datadog header, IP ranges and analytics filters.
3 min read
Daumoa and Daum/4.1: Daum Search Crawler User Agent
Daum's crawler: the Daum/4.1 and Daum-Preview user agents, the robots.txt token, the IP list, and what the older Daumoa name means.
4 min read
Diffbot Crawler: User Agents, robots.txt and Blocking
Diffbot's two user agents, Diffbot and Diffbot-User, what each one does, how Diffbot handles robots.txt and crawl-delay, and how to block them.
3 min read
Discordbot: Discord Link Preview User Agent and IPs
Discordbot fetches a page when someone shares its link in Discord, to build a preview card. Its user agent, IP list, and how to verify or block it.
2 min read