Baiduspider: Baidu Crawler User Agents and robots.txt
Baiduspider crawls the web for Baidu search. User agent tokens per Baidu product, reverse DNS verification and robots.txt rules from Baidu's help pages.
4 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 2
Guides to AI crawlers and bots that index your website for AI training and search.
Bingbot: Bing Crawler User Agent, robots.txt and Copilot
Bingbot crawls for Bing search and Copilot answers. User agent strings, DNS and IP verification, robots.txt rules, and noarchive and nocache AI controls.
4 min read
BingPreview: Bing Page Snapshot Crawler and User Agent
BingPreview makes page snapshots for Bing. Its current user agent carries the bingbot token. How to verify it and limit previews with meta tags.
3 min read
BuiltWith Bot: User Agent, robots.txt and Removal
BuiltWith visits sites to detect their technology stack. Its user agent values, the robots.txt rule it follows and how to remove your site profile.
3 min read
Bytespider: ByteDance Crawler User Agent and Blocking
Bytespider is the ByteDance crawler for Toutiao Search, reported as an AI training crawler. Its documented user agent, 2019 IP list and how to block it.
3 min read
CCBot: Common Crawl Crawler, User Agent and Blocking
CCBot collects pages for the Common Crawl web archive, a common AI training source. User agent, IP ranges, reverse DNS and robots.txt rules.
3 min read
ChatGLM-Spider: Zhipu AI Crawler and How to Block It
ChatGLM-Spider is a crawler that bot directories link to Zhipu AI. What is known, what is not documented, and how to block it.
2 min read
ChatGPT-User: OpenAI User Agent for ChatGPT Requests
ChatGPT-User visits pages when a ChatGPT user or custom GPT asks for them. User agent string, IP list, robots.txt status and server blocking.
4 min read
Claude-SearchBot: Anthropic Search Crawler and robots.txt
Claude-SearchBot indexes pages for search results in Claude. Its robots.txt token, what a block changes and how to verify requests.
3 min read