Google-InspectionTool: User Agent for Search Testing Tools
Google-InspectionTool user agents and robots.txt tokens. It fetches pages for URL Inspection and the Rich Results Test, not for Google Search.
2 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 5
Guides to AI crawlers and bots that index your website for AI training and search.
Googlebot: Google Search Crawler User Agent and robots.txt
Googlebot user-agent strings, robots.txt token, how to verify requests with reverse DNS, and what blocking Googlebot changes in Google Search.
4 min read
GoogleOther: Google Generic Crawler User Agent and Blocking
GoogleOther user agents, GoogleOther-Image and GoogleOther-Video, robots.txt rules, and why a block does not affect Google Search.
3 min read
GPTBot: OpenAI Crawler, User Agent and robots.txt
GPTBot user agent, robots.txt rules, IP ranges and what a block changes. Blocking GPTBot does not remove a site from ChatGPT search.
3 min read
GrapeshotCrawler: Grapeshot Bot User Agent and Status
GrapeshotCrawler was the contextual ad crawler of Oracle's Grapeshot. Oracle closed its ad business in 2024. User agent and how to block it.
2 min read
HubSpot Crawler: User Agent, IP Ranges and robots.txt
HubSpot Crawler fetches pages for HubSpot SEO tools, imports and the AI customer agent. Its user agent, IP ranges and robots.txt rules.
3 min read
ia_archiver: Alexa Crawler User Agent and robots.txt
ia_archiver is the crawler user agent of Alexa Internet, which closed in May 2022. Its history, what the old robots.txt group does today and how to handle it.
3 min read
ImagesiftBot: ImageSift Crawler User Agent and robots.txt
ImagesiftBot collects public images and page text for the ImageSift image index. Its user agent, robots.txt rules, Crawl-delay and opt-out.
2 min read
AI Crawler Bots: Training, Search and User Fetchers
What AI crawler bots are and their three types: training crawlers, AI search crawlers and user-triggered fetchers, with examples and sources.
3 min read