WappalyzerBot: Wappalyzer Crawler User Agent and robots.txt
WappalyzerBot crawls public pages to detect the technologies a site uses. Its user agent, signed requests, robots.txt rules and how to block it.
3 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 10
Guides to AI crawlers and bots that index your website for AI training and search.
Webzio-extended and Omgilibot: Webz.io Bots and robots.txt
Webzio-extended tags Webz.io crawl data as usable or not for AI training. What Webz.io documents about it and Omgilibot, and how to block its bots.
3 min read
WhatsApp Link Preview User Agent and Open Graph Tags
WhatsApp fetches a page to build a link preview. User-agent format, the Open Graph tags it reads, image limits, and how to block it.
4 min read
YandexBot: Yandex Crawler User Agent and robots.txt
YandexBot is the main Yandex search crawler. User-agent strings, robots.txt tokens, crawl rate, and the DNS check for real requests.
3 min read
YouBot: You.com Crawler User Agent and robots.txt
YouBot crawls pages for the You.com search engine. User-agent string, robots.txt rules, IP range, reverse DNS and Web Bot Auth checks.
3 min read