How to Track Brand Mentions in AI Search
Learn how to track brand mentions across ChatGPT, Perplexity, and Google AI Overviews using controlled prompts, citations, and clear KPIs.
12 min read
Eugene Mironichev
201 articles
AI Search Optimization: Guide to Answer Engines
Learn how AI search optimization helps content get cited in Google AI Overviews, ChatGPT Search, Perplexity, and Gemini.
11 min read
MCP Security Playbook for AI Agent Toolchains
Learn MCP security risks, threat models, governance checklists, and rollout advice for securing AI agent toolchains in 2026.
11 min read
Multi-Agent Coding: Production AI Workflows
Learn how multi-agent coding moves AI coding agents from vibe coding to controlled workflows with isolation, review queues, and cost controls.
14 min read
AI21 Jamba: Model Facts and Web Access for Site Owners
AI21 Jamba is a family of open language models with a 256K context window. It has no web access of its own, and AI21 documents no crawler.
2 min read
Amazon Q: Web Crawler, robots.txt and Site Access
What Amazon Q Developer and Amazon Q Business do with websites, and how to block the Amazon Q web crawler with the amazon-QBusiness token.
3 min read
BLOOM: BigScience Open Model, No Web Access
BLOOM is a 176B-parameter open multilingual model from BigScience, released in 2022. It has no web access or crawler, and its training data is fixed.
2 min read
Character.AI: Web Access and Crawlers for Site Owners
Character.AI documents no web search, page fetches or crawler. What this means for site owners, and why there is no robots.txt rule for it.
2 min read
Codeium, Now Windsurf and Devin Desktop: Web Access
Codeium became Windsurf in 2025 and Devin Desktop in 2026. How its agent searches and reads web pages, and why no robots.txt token exists.
2 min read
Cohere Command: RAG, Web Content and Citations
Cohere Command models have no web access. How apps pass web pages to them as documents, how Command cites those documents, and which Cohere bots exist.
3 min read
Copy.ai: Web Search, Page Scraping and Site Access
Copy.ai Chat searches the web and its workflows scrape pages. No user agent or robots.txt token is documented. What site owners can do.
2 min read
Cursor Agent: Web Search, Browser and Site Access
The Cursor agent can search the web, read links and control a browser. Cursor documents no user agent or robots.txt token for this traffic.
2 min read
Drift Chatbot: Salesloft Acquisition and Shutdown
Drift, the B2B chat product of Clari + Salesloft, has a gradual sunset with 1mind as successor. Dates, the 2025 breach and steps for sites.
3 min read
Duolingo Max: GPT-4 Features and Web Access
Duolingo Max is a Duolingo subscription tier with GPT-4 features. It describes no web search or crawler, so site owners have nothing to block.
2 min read
Falcon LLM by TII: Web Access and Common Crawl Data
Falcon open models from TII have no crawler. Their RefinedWeb training data comes from Common Crawl. How to opt out and request removal.
2 min read
Gemma: Google Open Models, Web Access and Training Data
Gemma is a family of open models from Google with no crawler or web access of its own. What site owners need to know about Gemma and web data.
2 min read
GitHub Copilot: Bing Web Search and Site Access
GitHub Copilot Chat uses Bing or model web search. GitHub documents no Copilot crawler or user agent. What site owners can control.
3 min read
Google Gemini: Web Access, Google-Extended and Sources
How Gemini uses web pages: grounding from the Google Search index, the Google-Extended robots.txt token, user-triggered fetchers and sources.
3 min read
GPT-NeoX: EleutherAI Open Model and Training Data
GPT-NeoX is the EleutherAI training library and 20B open model. It has no web access or crawler. Its training data, the Pile, includes Common Crawl text.
2 min read
Grammarly AI: Web Sources, Crawlers and Site Access
Grammarly, now part of Superhuman, checks text against web pages and suggests web sources. It documents no crawler or user agent. What owners can control.
2 min read
HuggingChat: Web Search, User Agents and Status
HuggingChat closed in July 2025 and returned in October 2025. How it searches the web through Exa, its URL fetcher user agent, and how to block both.
3 min read
IBM watsonx Assistant: Web Crawler and Search Sources
watsonx Assistant answers from content a business connects, not the open web. The Watson Discovery crawler user agent, citations and the Orchestrate move.
3 min read
Intercom Fin: Website Sync Crawler and robots.txt
Fin answers from content a business adds, not the open web. How Fin website sync crawls pages, why it has no user agent, and its robots.txt setting.
3 min read
Jasper Chat: How Jasper AI Reads Web Pages
Jasper Chat is a marketing assistant from Jasper AI. How Jasper reads URLs for brand voice and knowledge, cites sources, and what site owners can block.
2 min read
Khanmigo: Khan Academy AI Tutor and Web Access
Khanmigo is the Khan Academy AI tutor. It works inside Khan Academy pages with no documented web search or crawler. Khanmigo Lite runs in ChatGPT.
2 min read
Llama Models by Meta: Web Access and Training Data
Meta's Llama models have no web access of their own. How Llama web search tool calls work, what data trained them, and how to opt out of Meta crawling.
2 min read
Mistral Le Chat (Now Vibe): Web Search and User Agents
Mistral renamed Le Chat to Vibe in May 2026. How Vibe searches the web and cites sources, and how to allow or block its two user agents.
3 min read
Mistral AI Models: Licenses, Web Search and Training Bot
Mistral AI makes open-weight and API language models that do not browse the web. Licenses, the web search tool and the MistralAI-Training crawler.
3 min read
OpenChat: Open-Source LLM Fine-Tuned With C-RLFT
OpenChat is a family of open-source language models trained with C-RLFT. Versions, licenses, project status, and why it has no crawler or user agent.
2 min read
Microsoft Phi: Small Language Models With No Web Access
Phi is Microsoft's family of small language models under the MIT license. Current Phi-4 models, and why Phi has no crawler or user agent to block.
2 min read
Pi by Inflection AI: Status, Web Use and Site Owners
Pi is the personal AI from Inflection AI. Who runs it after the 2024 Microsoft hires, how it uses web sources, and why no Pi crawler is documented.
2 min read
Poe by Quora: Web Access and Site Owner Controls
Poe is Quora's app for chatting with many third-party AI bots. What is known about its web access, and why there is no Poe user agent to block.
2 min read
Qwen: Alibaba's Open-Weight Language Models
Qwen is Alibaba's family of language models, many with open weights. Current Qwen3.8 releases, licenses, and why Qwen has no crawler to block.
2 min read
Reka AI: Multimodal Models, Web Search and Site Owners
Reka AI builds multimodal models and video tools. Company status, what Reka documents about web search, and why there is no Reka crawler to block.
2 min read
Replika: AI Companion App, Web Access and Crawlers
Replika is an AI companion app from Luka, Inc. It does not search the web and has no documented crawler or user agent. What site owners need to know.
2 min read
Salesforce Einstein and Agentforce: Web Search and Crawlers
Einstein is Salesforce's built-in AI, and Agentforce replaced Einstein Copilot. Agentforce web search, website connectors and what site owners can block.
3 min read
StableLM: Stability AI Open Models and Web Access
StableLM is Stability AI's family of open-weight language models. Releases, licenses, web training data and why there is no StableLM bot to block.
2 min read
Tabnine: AI Code Assistant and Its Web Fetch Tool
Tabnine is an AI coding assistant. Its CLI has an optional Web Fetch tool that reads URLs a developer gives it. No user agent is documented.
3 min read
Tencent Hunyuan: AI Models, API Search and Web Access
Tencent Hunyuan models for text, images, video and 3D, the search and citation options of the Hunyuan API, and what site owners need to know.
2 min read
Tongyi Qianwen: Alibaba AI Assistant and Web Search
Tongyi Qianwen (通义千问) is the Chinese name of Alibaba's Qwen. The Qwen Chat and Qianwen assistants, their web search and what site owners can do.
3 min read
Woebot: Mental Health Chatbot App Retired in 2025
Woebot Health retired the Woebot app on June 30, 2025. What Woebot was, the data deadlines and why site owners have no Woebot bot to block.
2 min read
Writesonic Chatsonic: Web Search, URLs and Blocking
Chatsonic, the AI assistant from Writesonic, searches the web and reads URLs. Writesonic documents no user agent. What site owners can do.
2 min read
Wysa: Mental Health Chatbot and Web Access
Wysa is a mental health support chatbot from Wysa Ltd. How it writes replies, and why site owners have no Wysa bot to block.
2 min read
Yi by 01.AI: Open Bilingual Language Models
Yi is a family of open-weight English and Chinese language models from 01.AI. Releases, license and why site owners have no Yi bot to block.
2 min read
Zendesk AI: Web Crawler User Agent and Site Access
Zendesk AI agents answer from help centers and crawled sites. The Zendesk/External-Content crawler and how to allow or block it.
3 min read
360Spider: Qihoo 360 Crawler User Agent and Blocking
360Spider crawls the web for 360 Search (so.com). User agents, robots.txt tokens, the Indexpage extension, published IP ranges and blocking.
3 min read
Adbeat Crawler (adbeat_bot): User Agent and Blocking
adbeat_bot collects ad data for Adbeat, a display ad intelligence service. Its robots.txt token, user agent, and how to block it.
2 min read
AdIdxBot: Bing Ads Crawler User Agent and robots.txt
AdIdxBot is the Bing Ads crawler that checks ads and the pages they link to. User agent strings, robots.txt rules and how to verify requests.
3 min read
AhrefsBot: Ahrefs Crawler User Agent and robots.txt
AhrefsBot crawls for the Ahrefs SEO tools and the Yep search engine. User agent, IP lists, reverse DNS and robots.txt rules to block or slow it.
3 min read
AI2Bot: Ai2 Crawler User Agent and robots.txt
AI2Bot is the crawler of Ai2 (Allen Institute for AI). It collects content to train open models such as Olmo. User agent and how to block it.
3 min read
Amazonbot: Amazon Crawler User Agent and robots.txt
Amazonbot, Amzn-SearchBot and Amzn-User: what each Amazon agent does, user agent strings, IP lists, and how to block them in robots.txt.
3 min read
Applebot-Extended: Opt Out of Apple AI Training
Applebot-Extended is a robots.txt token, not a crawler. Disallow it to keep Apple from training its AI models on your pages. Apple search stays the same.
4 min read
Applebot: Apple Web Crawler, User Agent and robots.txt
Applebot crawls for Spotlight, Siri and Safari search and Apple AI features. User agent format, DNS and IP verification, robots.txt and meta tags.
4 min read
archive.org_bot: Internet Archive Crawler and robots.txt
archive.org_bot crawls pages for the Wayback Machine. Its user agent token, the limits of robots.txt, and how to request removal of archived pages.
3 min read
Baiduspider: Baidu Crawler User Agents and robots.txt
Baiduspider crawls the web for Baidu search. User agent tokens per Baidu product, reverse DNS verification and robots.txt rules from Baidu's help pages.
4 min read
Bingbot: Bing Crawler User Agent, robots.txt and Copilot
Bingbot crawls for Bing search and Copilot answers. User agent strings, DNS and IP verification, robots.txt rules, and noarchive and nocache AI controls.
4 min read
BingPreview: Bing Page Snapshot Crawler and User Agent
BingPreview makes page snapshots for Bing. Its current user agent carries the bingbot token. How to verify it and limit previews with meta tags.
3 min read
BuiltWith Bot: User Agent, robots.txt and Removal
BuiltWith visits sites to detect their technology stack. Its user agent values, the robots.txt rule it follows and how to remove your site profile.
3 min read
Bytespider: ByteDance Crawler User Agent and Blocking
Bytespider is the ByteDance crawler for Toutiao Search, reported as an AI training crawler. Its documented user agent, 2019 IP list and how to block it.
3 min read
CCBot: Common Crawl Crawler, User Agent and Blocking
CCBot collects pages for the Common Crawl web archive, a common AI training source. User agent, IP ranges, reverse DNS and robots.txt rules.
3 min read
ChatGLM-Spider: Zhipu AI Crawler and How to Block It
ChatGLM-Spider is a crawler that bot directories link to Zhipu AI. What is known, what is not documented, and how to block it.
2 min read
ChatGPT-User: OpenAI User Agent for ChatGPT Requests
ChatGPT-User visits pages when a ChatGPT user or custom GPT asks for them. User agent string, IP list, robots.txt status and server blocking.
4 min read
Claude-SearchBot: Anthropic Search Crawler and robots.txt
Claude-SearchBot indexes pages for search results in Claude. Its robots.txt token, what a block changes and how to verify requests.
3 min read
Claude-User: Anthropic Agent for User Requests
Claude-User opens a web page when a Claude user asks a question. Its robots.txt token, what a block changes and how to verify requests.
3 min read
ClaudeBot: Anthropic Crawler, User Agent and robots.txt
ClaudeBot is Anthropic's crawler for AI model training. Its robots.txt token, Anthropic's other bots, the IP list and how to block it.
3 min read
Cloudflare Always Online Crawler: User Agents and IPs
Cloudflare Always Online shows archived pages when your origin is down. The user agents and IP ranges that crawl your site, and how to turn it off.
3 min read
coccocbot: Coc Coc Crawler User Agents and robots.txt
coccocbot is the crawler family of the Coc Coc search engine. Its user agent strings, robots.txt rules, Crawl-delay limit and reverse DNS check.
3 min read
cohere-ai and cohere-training-data-crawler User Agents
cohere-ai and cohere-training-data-crawler are tokens that bot lists link to Cohere. Cohere documents neither. What Cohere says and how to block them.
3 min read
Datadog Synthetics Bot: User Agent, Headers and IPs
How to identify Datadog Synthetic Monitoring requests: the Datadog/Synthetics user agent, the sec-datadog header, IP ranges and analytics filters.
3 min read
Daumoa and Daum/4.1: Daum Search Crawler User Agent
Daum's crawler: the Daum/4.1 and Daum-Preview user agents, the robots.txt token, the IP list, and what the older Daumoa name means.
4 min read
Diffbot Crawler: User Agents, robots.txt and Blocking
Diffbot's two user agents, Diffbot and Diffbot-User, what each one does, how Diffbot handles robots.txt and crawl-delay, and how to block them.
3 min read
Discordbot: Discord Link Preview User Agent and IPs
Discordbot fetches a page when someone shares its link in Discord, to build a preview card. Its user agent, IP list, and how to verify or block it.
2 min read
DotBot: Moz Link Index Crawler and robots.txt Rules
DotBot is the crawler that builds the Moz Link Index. Its robots.txt token, how to block or slow it, and how it differs from Rogerbot.
3 min read
DuckAssistBot: DuckDuckGo Search Assist Crawler
DuckAssistBot fetches pages in real time for DuckDuckGo Search Assist answers. Its user agent, IP list, and the robots.txt opt-out.
2 min read
FacebookBot: Meta Crawler User Agent and Status
FacebookBot is not on Meta's current crawler list. What Meta documents now, which user agents to use in robots.txt, and how to block Meta crawlers.
3 min read
facebookexternalhit: Meta Link Preview Crawler
facebookexternalhit fetches shared links for previews on Facebook, Instagram and Messenger. Its user agents, robots.txt behavior and requirements.
4 min read
Feedfetcher-Google: Google Feed Fetcher and Blocking
FeedFetcher-Google fetches RSS and Atom feeds for Google News and WebSub. It ignores robots.txt. User agent, IP list and how to block it.
3 min read
Feedly Fetcher (FeedlyBot): User Agent and Blocking
Feedly Fetcher, also called FeedlyBot, gets RSS feeds for Feedly users as Feedly/1.0. It ignores robots.txt. How to block it on the server.
3 min read
FriendlyCrawler: Unknown AI Dataset Crawler and Blocking
FriendlyCrawler has no known operator. Bot lists say it builds machine learning datasets. What is known and how to block it in robots.txt.
2 min read
Google-CloudVertexBot: Vertex AI Crawler and robots.txt
Google-CloudVertexBot crawls sites for Vertex AI search (Agent Search) data stores. User agent, robots.txt tokens, sitemaps and verification.
3 min read
Google-Extended: robots.txt Token for Gemini Training
Google-Extended is a robots.txt token, not a crawler. It controls Gemini training and grounding, not Google Search ranking or AI Overviews.
3 min read
Google-InspectionTool: User Agent for Search Testing Tools
Google-InspectionTool user agents and robots.txt tokens. It fetches pages for URL Inspection and the Rich Results Test, not for Google Search.
2 min read
Googlebot: Google Search Crawler User Agent and robots.txt
Googlebot user-agent strings, robots.txt token, how to verify requests with reverse DNS, and what blocking Googlebot changes in Google Search.
4 min read
GoogleOther: Google Generic Crawler User Agent and Blocking
GoogleOther user agents, GoogleOther-Image and GoogleOther-Video, robots.txt rules, and why a block does not affect Google Search.
3 min read
GPTBot: OpenAI Crawler, User Agent and robots.txt
GPTBot user agent, robots.txt rules, IP ranges and what a block changes. Blocking GPTBot does not remove a site from ChatGPT search.
3 min read
GrapeshotCrawler: Grapeshot Bot User Agent and Status
GrapeshotCrawler was the contextual ad crawler of Oracle's Grapeshot. Oracle closed its ad business in 2024. User agent and how to block it.
2 min read
HubSpot Crawler: User Agent, IP Ranges and robots.txt
HubSpot Crawler fetches pages for HubSpot SEO tools, imports and the AI customer agent. Its user agent, IP ranges and robots.txt rules.
3 min read
ia_archiver: Alexa Crawler User Agent and robots.txt
ia_archiver is the crawler user agent of Alexa Internet, which closed in May 2022. Its history, what the old robots.txt group does today and how to handle it.
3 min read
ImagesiftBot: ImageSift Crawler User Agent and robots.txt
ImagesiftBot collects public images and page text for the ImageSift image index. Its user agent, robots.txt rules, Crawl-delay and opt-out.
2 min read
AI Crawler Bots: Training, Search and User Fetchers
What AI crawler bots are and their three types: training crawlers, AI search crawlers and user-triggered fetchers, with examples and sources.
3 min read
How to Find and Block AI Crawlers on Your Site
Find AI crawlers in your access log, match their IP addresses to operator lists, and control them with robots.txt and firewall rules.
5 min read
Kangaroo Bot: Kangaroo LLM Crawler and robots.txt
Kangaroo Bot is the crawler of Kangaroo LLM, an Australian open-source LLM project. Its robots.txt token, the project status and how to block it.
3 min read
LinkedInBot: LinkedIn Link Preview Bot User Agent
LinkedInBot reads a page when someone shares its URL on LinkedIn, to build the link preview. User agents, Open Graph tags and robots.txt.
3 min read
Meta-ExternalAgent: Meta AI Crawler and robots.txt
Meta-ExternalAgent crawls the web to train Meta AI models and index content. User agent, robots.txt rules and what a block changes.
3 min read
Meta-ExternalFetcher: User Agent and How to Block It
Meta-ExternalFetcher fetches single links when a user asks a Meta product. It can bypass robots.txt. User agent and server blocking.
3 min read
MJ12bot: Majestic Crawler User Agent and robots.txt
MJ12bot is the Majestic backlink crawler. It obeys robots.txt and Crawl-Delay up to 20 seconds. How to block, slow and identify it.
3 min read
MLBot: Undocumented Crawler User Agent and Blocking
MLBot has no operator documentation. What public bot lists say, what is unknown, and how to block it with robots.txt and nginx.
2 min read
MojeekBot: Mojeek Crawler User Agent and robots.txt
MojeekBot crawls for the Mojeek search engine. robots.txt token, noindex support, crawl rate and how to identify real requests.
2 min read
MSNBot: Retired Microsoft Crawler Replaced by Bingbot
MSNBot was renamed Bingbot on October 1, 2010. Old msnbot robots.txt rules, msnbot host names, and how to identify Bing requests.
2 min read
Naverbot (Yeti): Naver Crawler User Agent and robots.txt
Naver calls its search crawler Yeti. User-agent strings, robots.txt examples, HTTP status handling and how to identify real requests.
3 min read
New Relic Synthetics: Monitor Traffic, Headers and IPs
New Relic synthetic monitors are scheduled checks, not crawlers. Identify them by the X-Abuse-Info header and New Relic IP ranges.
3 min read
OAI-SearchBot: OpenAI Search Crawler and robots.txt
OAI-SearchBot finds pages for ChatGPT search. User-agent string, robots.txt rules, how it differs from GPTBot, and OpenAI IP ranges.
3 min read
PanguBot: Huawei AI Crawler, What Is Known and Blocking
Third-party bot lists link PanguBot to Huawei PanGu AI training, but Huawei does not document it. How to block it in robots.txt and nginx.
2 min read
Perplexity-User: User Agent for Perplexity Page Fetches
Perplexity-User fetches pages when a user asks Perplexity a question and generally ignores robots.txt. User agent, IP list and server blocking.
3 min read
PerplexityBot: Perplexity Search Crawler and robots.txt
PerplexityBot finds and links sites for Perplexity search results. User agent, IP list, robots.txt rules and firewall setup.
3 min read
PetalBot: Huawei Petal Search Crawler and robots.txt
PetalBot crawls for Huawei Petal Search and AI Search. User agents, robots.txt rules, reverse DNS check and what a block changes.
3 min read
Pingdom Bot: Uptime Check Requests, IPs and Blocking
Pingdom probe servers request URLs that Pingdom users monitor. Check types, published probe IP lists, and how to allow or block them.
3 min read
Pinterestbot: Pinterest Crawler User Agent and robots.txt
Pinterestbot indexes pages and updates Pin details. User agents, robots.txt and crawl-delay rules, the nopin tag and reverse DNS checks.
3 min read
Proximic: Comscore Ad Crawler User Agent and robots.txt
The proximic crawler analyzes pages for Comscore contextual advertising. User agent, robots.txt rules, crawl delay and what a block changes.
3 min read
Rogerbot: Moz Site Audit Crawler and robots.txt
Rogerbot is the Moz Pro site audit crawler, separate from the Dotbot link crawler. How to allow, block or slow it in robots.txt and how to identify it.
3 min read
Screaming Frog SEO Spider: User Agent and robots.txt
Screaming Frog SEO Spider is a desktop crawler that people run on their own computers. Its user agent, robots.txt handling and how to block it.
3 min read
SecurityTrails Bot: What Is Known and How to Block It
SecurityTrails sells DNS, WHOIS and IP data to security teams. What is documented about a SecurityTrails bot and how to handle its requests.
2 min read
SemrushBot: Semrush Crawler User Agents and robots.txt
SemrushBot and the other Semrush bots collect data for Semrush SEO tools. Their robots.txt tokens, Crawl-delay limit and how to block each one.
3 min read
SeobilityBot: Seobility SEO Crawler and robots.txt
SeobilityBot crawls sites for the Seobility SEO tools. Its user agent, IP list and robots.txt rules to block it or to let only it crawl a site.
2 min read
Serpstatbot: Serpstat Backlink Crawler and robots.txt
Serpstatbot crawls the web for the Serpstat backlink index. Its robots.txt token, Crawl-delay limit, IP list and why a robots.txt block can fail.
3 min read
SeznamBot: Seznam.cz Crawler User Agent and robots.txt
SeznamBot crawls for Seznam.cz search. Its user agent, IP list, reverse DNS names and robots.txt rules, including the Request-rate directive.
4 min read
SISTRIX Crawler: User Agent, Reverse DNS and robots.txt
The SISTRIX Crawler fetches pages for the SISTRIX SEO Toolbox. Its default user agent, reverse DNS names, robots.txt rules and project settings.
3 min read
Slackbot: Slack Link Preview User Agents and Blocking
Slack fetches links that users post in Slack to build previews. Its user agents, why it ignores robots.txt, the tags it reads and how to block it.
3 min read
Snapchat URL Preview Service: User Agent and Blocking
Snapchat fetches shared links with the Snap URL Preview Service to build chat previews. Its user agent, cache time, tags and how to block it.
3 min read
Sogou web spider: Sogou Search Crawler and robots.txt
Sogou web spider crawls pages for Sogou, a Chinese search engine. Its user agent, robots.txt support, reverse DNS check and how to block it.
3 min read
Storebot-Google: Google Shopping Crawler and robots.txt
Storebot-Google crawls product, cart and checkout pages for Google Shopping. User agents, robots.txt token, verification and what a block changes.
3 min read
TelegramBot: Telegram Link Preview Fetcher User Agent
TelegramBot fetches shared links for link previews and Instant View in Telegram. What Telegram documents, the known user agent and how to block it.
2 min read
TikTokSpider: TikTok Crawler User Agent and Blocking
TikTokSpider names TikTok in its user agent, but TikTok publishes no docs for it. The known user agent, what is not confirmed and how to block it.
2 min read
Timpibot: Timpi Crawler User Agent, Blocking and Removal
Timpibot crawls the web for Timpi Search and the Timpi Data API. Known user agent, robots.txt status, takedown requests and how to block it.
3 min read
Twitterbot: X Link Preview Crawler User Agent
Twitterbot fetches shared links for X (Twitter) card previews. X no longer documents it. The known user agent, card tags and how to block it.
2 min read
UptimeRobot: Monitoring Bot User Agent and IP List
UptimeRobot checks URLs that its users monitor. Its user agent, published IP lists, HEAD requests and how to allow or block the checks.
3 min read
WappalyzerBot: Wappalyzer Crawler User Agent and robots.txt
WappalyzerBot crawls public pages to detect the technologies a site uses. Its user agent, signed requests, robots.txt rules and how to block it.
3 min read
Webzio-extended and Omgilibot: Webz.io Bots and robots.txt
Webzio-extended tags Webz.io crawl data as usable or not for AI training. What Webz.io documents about it and Omgilibot, and how to block its bots.
3 min read
WhatsApp Link Preview User Agent and Open Graph Tags
WhatsApp fetches a page to build a link preview. User-agent format, the Open Graph tags it reads, image limits, and how to block it.
4 min read
YandexBot: Yandex Crawler User Agent and robots.txt
YandexBot is the main Yandex search crawler. User-agent strings, robots.txt tokens, crawl rate, and the DNS check for real requests.
3 min read
YouBot: You.com Crawler User Agent and robots.txt
YouBot crawls pages for the You.com search engine. User-agent string, robots.txt rules, IP range, reverse DNS and Web Bot Auth checks.
3 min read
Algolia: Hosted Site Search and the Algolia Crawler
Algolia is hosted search that companies add to their own sites, not a web search engine. Algolia Crawler user agent, robots.txt and IP.
3 min read
Andi Search: AI Search Engine Facts for Site Owners
Andi is a Y Combinator-backed AI search engine that answers with sources. What is known about it, and what it does not document for sites.
2 min read
Arc Search: AI Browser App Shutting Down in 2026
Arc Search, the AI mobile browser from The Browser Company, is shutting down. Key dates, what Browse for Me did, and what it means for sites.
2 min read
Baidu ERNIE and ERNIE Bot: AI Search and Baiduspider
Baidu uses ERNIE models in the ERNIE Bot app and for AI answers in Baidu Search. Which crawler collects pages, and the robots.txt controls.
3 min read
Brave Search AI Answers: Crawler, Index and Opt-Out
Brave Search uses its own index for AI Answers and Ask Brave. How the Brave crawler follows Googlebot rules and how to remove a page.
4 min read
CoCounsel: Thomson Reuters Legal AI, Formerly Casetext
CoCounsel is a legal AI assistant from Casetext, now part of Thomson Reuters. It is not a public web search engine. Key dates and facts.
2 min read
ChatGPT Search: How It Finds and Cites Web Pages
How ChatGPT search finds and cites web pages, which OpenAI bots to allow or block, how to opt out, and how ChatGPT visits show in analytics.
4 min read
ChatGPT Shopping: Product Feeds, Checkout and Opt-Out
How products appear in ChatGPT shopping: OAI-SearchBot, OpenAI product feeds for approved merchants, feed flags and Instant Checkout.
4 min read
Claude Web Search and Web Fetch: How Claude Cites Pages
How Claude finds, reads and cites web pages in its apps and in the API, which Anthropic bots to allow, and how to keep search but block training.
3 min read
Connected Papers: Data Source and How Graphs Are Built
Connected Papers builds similarity graphs of research papers from the Semantic Scholar corpus. How the graph works and how papers get in.
3 min read
Consensus: AI Search for Research Papers and Sources
Consensus searches over 200 million research papers from Semantic Scholar, OpenAlex and its own crawl. How its search and ranking work.
2 min read
Coveo: Enterprise Search Software and Coveobot Crawler
Coveo is search software that companies run on their own sites. How its crawler Coveobot handles robots.txt, noindex and crawl rate.
3 min read
DeepSeek: Chat Web Search, Search API and Fetch API
DeepSeek runs its own search engine for its chat app and its Search and Fetch APIs. What is documented, and what site owners can control.
3 min read
DuckDuckGo AI Chat (Duck.ai) and Search Assist
Duck.ai gives anonymous access to third-party AI models. How Search Assist answers cite pages, and which DuckDuckGo crawlers to allow.
3 min read
Elasticsearch: Self-Hosted Search and the Elastic Crawler
Elasticsearch is search software you run yourself or on Elastic Cloud. Its semantic search options and the Elastic Open Crawler user agent.
3 min read
Elicit: AI Research Assistant and Its Paper Sources
Elicit searches about 138 million papers from Semantic Scholar, OpenAlex and PubMed. Its sources, search method and web access.
3 min read
Exa AI: Search API, ExaSearchBot Crawler and Opt-Out
Exa is a web search API for AI apps. Its crawler ExaSearchBot follows robots.txt and signs requests. User agents, opt-out and removal.
3 min read
Felo AI: Multilingual AI Search and Web Access
Felo is a multilingual AI search engine from Felo Inc. It answers from real-time web results with citations. No crawler is documented.
2 min read
Glean: Enterprise AI Search and Its Website Crawler
Glean searches a company's own apps, not the public web. How its Website connector crawls sites and follows robots.txt, and how its web search works.
3 min read
Google AI Overviews: How Pages Appear and Opt-Out Controls
How Google AI Overviews pick and link pages, what a page needs to appear, and which controls apply: nosnippet, max-snippet and noindex.
4 min read
Grok by xAI: Chatbot, Web Search, Citations, Site Access
Grok is xAI's AI assistant. How it searches the web and X, how it cites sources, and why xAI documents no crawler or robots.txt token.
3 min read
Harvey AI: Legal AI Platform, Sources and Site Access
Harvey is AI software for law firms, not a public search engine. Its web search and legal content sources, and what site owners can control.
2 min read
iAsk.Ai: AI Answer Engine, Sources and Site Access
iAsk.Ai is a free AI answer engine. What it documents about its web index, citations, crawler and robots.txt access for site owners.
2 min read
Kagi Search: Kagibot User Agent, Sources and robots.txt
Kagi builds its own index with Kagibot. Kagibot user agent, IP addresses, robots.txt rules with Googlebot fallback, and Kagi Small Web.
3 min read
Komo AI Search: Sources, Status and Site Access
Komo is an AI search engine that answers with sources. Its status in October 2026, how it shows sources, and what it documents for site owners.
2 min read
Lexis+ AI (Now Lexis+ with Protégé): Sources and Access
Lexis+ AI, renamed Lexis+ with Protégé in February 2026, answers from LexisNexis legal content. Its sources and what site owners can control.
2 min read
Liner AI Search: Sources, Citations and Site Access
Liner is an AI search and research tool with web and scholar search. Which sites it cites most, and what it documents about crawler access.
2 min read
MediSearch: AI Medical Search Engine and Its Sources
MediSearch answers medical questions from selected scientific sources, not the open web. How it picks and cites sources, and what site owners can do.
2 min read
Meta AI: Chat, Web Search, Crawlers and Opt-Out
Meta AI chat and search use Bing, Google and the meta-webindexer crawler. How Meta AI cites pages and how to allow or block Meta crawlers.
3 min read
Microsoft Copilot: Chat, Bing Search and Citations
Microsoft Copilot answers web questions with Bing results and cites pages. How to get cited, opt out with noarchive or nocache, and track citations.
4 min read
Mojeek: Independent Search Index and AI Summary
Mojeek is a UK search engine with its own index. Who runs it, how its AI Search Summary cites results, and its NoML proposal for machine learning.
2 min read
Naver Cue: AI Search Shut Down, Now AI Briefing
Naver Cue: was the generative AI search of Naver. It shut down in April 2026. Naver AI answers now appear as AI Briefing in Naver search.
2 min read
OpenEvidence: Medical AI Search and Its Sources
OpenEvidence answers clinical questions from licensed journals such as NEJM and JAMA, with citations. It documents no web crawler.
2 min read
Perplexity AI: Search, Chatbot, Comet and Opt-Out
Perplexity answers questions with links to the pages it used. Its search, chatbot, Comet browser, publisher payments, APIs and site opt-out.
3 min read
Phind: AI Search Engine for Developers, Shut Down 2026
Phind was an AI search engine for developers from a Y Combinator company. It shut down in January 2026. No robots.txt rules are needed.
1 min read
PimEyes: Face Search Engine, PimEyesBot and Opt-Out
PimEyes is a face search engine that indexes faces from public web pages. How it finds images, what PimEyesBot does, and how to opt out.
3 min read
Qwant: French Search Engine, QwantBot and AI Answers
Qwant builds its index with QwantBot and cites pages in its AI answers. How to appear, block or slow QwantBot, and identify real requests.
4 min read
ResearchRabbit: Paper Discovery Tool and Data Sources
ResearchRabbit finds academic papers from Crossref, Semantic Scholar and OpenAlex data. It does not crawl websites. What publishers can do.
2 min read
SciSpace: AI Research Assistant and Paper Search
SciSpace by PubGenius Inc. is an AI research assistant with a search over 280M+ papers and PDF tools. It documents no crawler for site owners to block.
2 min read
Scite: Smart Citations, Data Sources and Publishers
Scite reads full-text papers from publisher agreements and open access content and labels citations as supporting, contrasting or mentioning.
3 min read
SearXNG: Open-Source Metasearch Engine and Your Site
SearXNG is open-source metasearch software that queries other search engines. It has no index or crawler, so your visibility depends on those engines.
2 min read
Semantic Scholar: Ai2 Paper Search and Its Sources
Semantic Scholar is the free research paper search engine of Ai2. Where its papers come from, how authors fix or remove papers, and how AI2Bot differs.
3 min read
Startpage: Private Search With Google and Bing Results
Startpage sends queries to Google and Bing without personal data. It has no index or crawler, so Google and Bing decide what it shows for your site.
2 min read
Tavily: Search API for AI Agents and Site Owners
Tavily is a search, extract and crawl API for AI agents, now joining Nebius. It documents no crawler user agent or robots.txt token.
2 min read
TinEye: Reverse Image Search, Crawler and Image Removal
TinEye finds where an image appears online with its own crawler and image index. What it documents about crawling and how to remove your images.
3 min read
Undermind: AI Literature Search on Semantic Scholar
Undermind is an AI research agent that searches the Semantic Scholar database of 200M+ papers and writes cited reports. It documents no crawler.
2 min read
Westlaw Edge AI: Status, Newer Versions and Sources
Westlaw Edge is a Thomson Reuters legal research platform from 2018, still sold next to Westlaw Advantage. It uses Westlaw content, not the open web.
2 min read
Wolfram|Alpha: Computed Answers, Not Web Search
Wolfram|Alpha computes answers from the Wolfram Knowledgebase, its own edited data, and does not crawl the web. What this means for site owners.
2 min read
YandexGPT in Yandex Search: AI Answers and Opt-Out
How YandexGPT and Yandex AI answers use indexed pages as sources, and how to opt out with the YandexAdditional robots.txt rule.
4 min read
Yep: Ahrefs Search Engine, Search API and AhrefsBot
Yep is the Ahrefs search engine, now focused on a web search API. AhrefsBot builds its index. How to opt out or submit pages with IndexNow.
3 min read
You.com: AI Search APIs, Citations and Site Access
You.com sells web search and answer APIs for AI apps. What each API returns, how cited answers show sources, and which crawler builds the index.
3 min read
Bing Webmaster Tools & IndexNow Setup Guide
Complete guide to Bing Webmaster Tools and IndexNow protocol. Learn API setup, URL submission methods, WordPress plugins, and Cloudflare integration.
14 min read
Common Crawl Presence Guide: Check Your Site in AI Data
Learn how to check if your website is in Common Crawl's archive, understand CCBot crawling cycles, and decide if presence benefits your business.
7 min read
Google Search Console for AI Visibility Tracking Guide
Learn how to use Google Search Console to monitor AI visibility. Setup GSC, track AI traffic patterns, and control content in AI Overviews.
9 min read
JSON-LD Schema Markup for AI Visibility Guide
Learn how to make content machine-readable for AI systems using JSON-LD schema markup. Includes code examples, validation tools, and implementation tips.
12 min read
llms.txt Specification Guide: Format, Implementation & Reality
Complete guide to llms.txt specification, format structure, implementation tools, and realistic adoption expectations for AI documentation optimization.
11 min read
Robots.txt for AI Crawlers: Complete Guide
Learn how to control AI crawler access with robots.txt. Block GPTBot, ClaudeBot, CCBot and other AI bots. Includes exact syntax and validation methods.
11 min read
Wikipedia & AI Visibility: What Businesses Need to Know
How Wikipedia shapes AI knowledge and why most businesses won't qualify for a page. Learn notability rules and AI training facts.
11 min read
Canonical Tag: rel=canonical Syntax and How Google Uses It
The rel=canonical link names the preferred URL for duplicate pages. Syntax, HTTP header form, how Google and Yandex use it, and its limits.
4 min read
Meta Author and Generator Tags: Syntax and Search Use
Meta author and meta generator tags name a page author and the software that built it. Syntax, HTML rules, and why Google ignores both.
3 min read
Meta Charset: Declare UTF-8 Character Encoding in HTML
The meta charset tag declares the character encoding of an HTML page. Syntax, HTML standard rules, HTTP header precedence and what it does not do.
3 min read
Meta Description Tag: Syntax and How Google Uses It
The meta description tag gives a short page summary. Syntax, HTML rules, and how Google and Yandex use it for snippets or replace it with page text.
4 min read
Meta Keywords Tag: Syntax and How Search Engines Treat It
Syntax of the meta keywords tag, what the HTML standard says about it, and why Google Search does not use it for indexing or ranking.
3 min read
Robots Meta Tag: noindex, nosnippet and Other Rules
Robots meta tag and X-Robots-Tag syntax, the rules Google, Bing and Apple document, and how nosnippet affects Google AI Overviews.
5 min read
HTML Meta Tags Cheat Sheet: Syntax and What Each Tag Does
Compact table of HTML meta tags, Open Graph and X card tags: what each tag does, what Google supports or ignores, and links to full guides.
4 min read
Meta Viewport Tag: Syntax, Values and Zoom Settings
Viewport meta tag syntax, every content value, what Google says about it, and why you must not turn off zoom with user-scalable=no.
3 min read
og:image Meta Tag: Syntax, Sizes and How Sites Use It
og:image syntax and image properties, the size rules Facebook and LinkedIn document, and how Google uses og:image for preview images.
4 min read
Open Graph Meta Tags: og:title, og:type, og:url, og:image
Open Graph protocol syntax, the four required og tags, optional tags and types, and how Facebook, LinkedIn and Google use them.
3 min read
Twitter Card Meta Tags: twitter:card Syntax and Card Types
twitter:card syntax, card types, image limits and Open Graph fallback, from the last published X Cards docs. X no longer publishes them.
4 min read
Track AI Assistant Traffic in GA4 with UTM Tags
How GA4 reports visits from ChatGPT and other AI assistants, why AI Overviews count as Organic Search, and how to tag your own links in llms.txt.
5 min read