Naverbot (Yeti): Naver Crawler User Agent and robots.txt
Naver calls its search crawler Yeti. User-agent strings, robots.txt examples, HTTP status handling and how to identify real requests.
3 min read
AI CRAWLER BOTS
AI Crawler Bots - Page 7
Guides to AI crawlers and bots that index your website for AI training and search.
New Relic Synthetics: Monitor Traffic, Headers and IPs
New Relic synthetic monitors are scheduled checks, not crawlers. Identify them by the X-Abuse-Info header and New Relic IP ranges.
3 min read
OAI-SearchBot: OpenAI Search Crawler and robots.txt
OAI-SearchBot finds pages for ChatGPT search. User-agent string, robots.txt rules, how it differs from GPTBot, and OpenAI IP ranges.
3 min read
PanguBot: Huawei AI Crawler, What Is Known and Blocking
Third-party bot lists link PanguBot to Huawei PanGu AI training, but Huawei does not document it. How to block it in robots.txt and nginx.
2 min read
Perplexity-User: User Agent for Perplexity Page Fetches
Perplexity-User fetches pages when a user asks Perplexity a question and generally ignores robots.txt. User agent, IP list and server blocking.
3 min read
PerplexityBot: Perplexity Search Crawler and robots.txt
PerplexityBot finds and links sites for Perplexity search results. User agent, IP list, robots.txt rules and firewall setup.
3 min read
PetalBot: Huawei Petal Search Crawler and robots.txt
PetalBot crawls for Huawei Petal Search and AI Search. User agents, robots.txt rules, reverse DNS check and what a block changes.
3 min read
Pingdom Bot: Uptime Check Requests, IPs and Blocking
Pingdom probe servers request URLs that Pingdom users monitor. Check types, published probe IP lists, and how to allow or block them.
3 min read
Pinterestbot: Pinterest Crawler User Agent and robots.txt
Pinterestbot indexes pages and updates Pin details. User agents, robots.txt and crawl-delay rules, the nopin tag and reverse DNS checks.
3 min read