Slackbot: Slack Link Preview User Agents and Blocking
Slack fetches links that users post in Slack to build previews. Its user agents, why it ignores robots.txt, the tags it reads and how to block it.
3 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 9
Guides to AI crawlers and bots that index your website for AI training and search.
Snapchat URL Preview Service: User Agent and Blocking
Snapchat fetches shared links with the Snap URL Preview Service to build chat previews. Its user agent, cache time, tags and how to block it.
3 min read
Sogou web spider: Sogou Search Crawler and robots.txt
Sogou web spider crawls pages for Sogou, a Chinese search engine. Its user agent, robots.txt support, reverse DNS check and how to block it.
3 min read
Storebot-Google: Google Shopping Crawler and robots.txt
Storebot-Google crawls product, cart and checkout pages for Google Shopping. User agents, robots.txt token, verification and what a block changes.
3 min read
TelegramBot: Telegram Link Preview Fetcher User Agent
TelegramBot fetches shared links for link previews and Instant View in Telegram. What Telegram documents, the known user agent and how to block it.
2 min read
TikTokSpider: TikTok Crawler User Agent and Blocking
TikTokSpider names TikTok in its user agent, but TikTok publishes no docs for it. The known user agent, what is not confirmed and how to block it.
2 min read
Timpibot: Timpi Crawler User Agent, Blocking and Removal
Timpibot crawls the web for Timpi Search and the Timpi Data API. Known user agent, robots.txt status, takedown requests and how to block it.
3 min read
Twitterbot: X Link Preview Crawler User Agent
Twitterbot fetches shared links for X (Twitter) card previews. X no longer documents it. The known user agent, card tags and how to block it.
2 min read
UptimeRobot: Monitoring Bot User Agent and IP List
UptimeRobot checks URLs that its users monitor. Its user agent, published IP lists, HEAD requests and how to allow or block the checks.
3 min read