Duolingo Max: GPT-4 Features and Web Access
Duolingo Max is a Duolingo subscription tier with GPT-4 features. It describes no web search or crawler, so site owners have nothing to block.
2 min read
AI CHAT BOT
Ai Chat Bot - Page 2
Falcon LLM by TII: Web Access and Common Crawl Data
Falcon open models from TII have no crawler. Their RefinedWeb training data comes from Common Crawl. How to opt out and request removal.
2 min read
Gemma: Google Open Models, Web Access and Training Data
Gemma is a family of open models from Google with no crawler or web access of its own. What site owners need to know about Gemma and web data.
2 min read
GitHub Copilot: Bing Web Search and Site Access
GitHub Copilot Chat uses Bing or model web search. GitHub documents no Copilot crawler or user agent. What site owners can control.
3 min read
Google Gemini: Web Access, Google-Extended and Sources
How Gemini uses web pages: grounding from the Google Search index, the Google-Extended robots.txt token, user-triggered fetchers and sources.
3 min read
GPT-NeoX: EleutherAI Open Model and Training Data
GPT-NeoX is the EleutherAI training library and 20B open model. It has no web access or crawler. Its training data, the Pile, includes Common Crawl text.
2 min read
Grammarly AI: Web Sources, Crawlers and Site Access
Grammarly, now part of Superhuman, checks text against web pages and suggests web sources. It documents no crawler or user agent. What owners can control.
2 min read
HuggingChat: Web Search, User Agents and Status
HuggingChat closed in July 2025 and returned in October 2025. How it searches the web through Exa, its URL fetcher user agent, and how to block both.
3 min read
IBM watsonx Assistant: Web Crawler and Search Sources
watsonx Assistant answers from content a business connects, not the open web. The Watson Discovery crawler user agent, citations and the Orchestrate move.
3 min read