Adbeat Crawler (adbeat_bot): User Agent and Blocking
adbeat_bot collects ad data for Adbeat, a display ad intelligence service. Its robots.txt token, user agent, and how to block it.
Adbeat is a competitive intelligence service for display advertising. Its crawler, adbeat_bot, visits websites to record the ads that they show. Adbeat says that the crawler follows robots.txt, so you can block it with the token adbeat_bot.
What the crawler collects
The Adbeat operation policy describes the crawler:
- Adbeat collects a statistical sample of the ads on the web.
- Its customers use the data to decide how and where to advertise.
- Adbeat crawls a site when it thinks the site has ad space that interests its customers.
- Adbeat says that it does not take content from your site. It collects only ad statistics.
The policy page does not mention AI training or search indexing.
User agent and robots.txt token
The robots.txt token is adbeat_bot. The Adbeat policy page does not show the full user agent string. Known Agents, a third-party bot directory, lists this string:
+https://www.adbeat.com/policy adbeat_botThe /policy address in that string now opens an error page. The policy is at /operation_policy.
Block adbeat_bot in robots.txt
If you want to block the crawler from your whole site, add this group. Adbeat gives the same example in its policy:
User-agent: adbeat_bot
Disallow: /If you want to block only one part of the site, change the path in the Disallow line:
User-agent: adbeat_bot
Disallow: /blog/Do not use robots meta tags for this crawler. Adbeat says that it ignores tags such as <meta name="robots" content="noindex, nofollow">.
A group for adbeat_bot applies only to this crawler. Search engine crawlers use their own tokens, so this rule does not change your search visibility.
Verify requests
Adbeat does not publish IP addresses or a reverse DNS name for its crawler. You can only match the user agent, and any client can send the same string.
If a client with this user agent ignores your robots.txt, block it in your web server or firewall. Match the text adbeat_bot in the user agent header. For crawler problems, Adbeat gives a contact address on its policy page.