Intercom Fin: Website Sync Crawler and robots.txt
Fin answers from content a business adds, not the open web. How Fin website sync crawls pages, why it has no user agent, and its robots.txt setting.
3 min read
AI CHAT BOT
Ai Chat Bot - Page 3
Jasper Chat: How Jasper AI Reads Web Pages
Jasper Chat is a marketing assistant from Jasper AI. How Jasper reads URLs for brand voice and knowledge, cites sources, and what site owners can block.
2 min read
Khanmigo: Khan Academy AI Tutor and Web Access
Khanmigo is the Khan Academy AI tutor. It works inside Khan Academy pages with no documented web search or crawler. Khanmigo Lite runs in ChatGPT.
2 min read
Llama Models by Meta: Web Access and Training Data
Meta's Llama models have no web access of their own. How Llama web search tool calls work, what data trained them, and how to opt out of Meta crawling.
2 min read
Mistral Le Chat (Now Vibe): Web Search and User Agents
Mistral renamed Le Chat to Vibe in May 2026. How Vibe searches the web and cites sources, and how to allow or block its two user agents.
3 min read
Mistral AI Models: Licenses, Web Search and Training Bot
Mistral AI makes open-weight and API language models that do not browse the web. Licenses, the web search tool and the MistralAI-Training crawler.
3 min read
OpenChat: Open-Source LLM Fine-Tuned With C-RLFT
OpenChat is a family of open-source language models trained with C-RLFT. Versions, licenses, project status, and why it has no crawler or user agent.
2 min read
Microsoft Phi: Small Language Models With No Web Access
Phi is Microsoft's family of small language models under the MIT license. Current Phi-4 models, and why Phi has no crawler or user agent to block.
2 min read
Pi by Inflection AI: Status, Web Use and Site Owners
Pi is the personal AI from Inflection AI. Who runs it after the 2024 Microsoft hires, how it uses web sources, and why no Pi crawler is documented.
2 min read