Proximic: Comscore Ad Crawler User Agent and robots.txt

Proximic is the name in the user agent of the Comscore crawler. Comscore uses it to analyze page content for advertising, so that its partners can match ads to the topic of a page. Comscore says that the crawler honors robots.txt, so you can allow, limit or block it with a proximic group.

What the crawler does

Comscore’s Web Crawler page says that its contextual analysis helps advertising partners find the best campaign for the content of a page. The crawler usually visits a page when an ad is about to be served there. Comscore can also crawl a whole site to give site-level analysis to advertising partners.

Comscore states that the crawler does not extract or store source code. It records data about the public content of a page, such as:

• The content language
• A content rating (G, PG13 or R)
• The IAB categories of the content, for example “Real Estate::Buying/Selling Homes”

IAB categories are a standard list of ad content topics. Comscore also says that the crawler downloads only static text and does not run JavaScript to generate ad impressions.

User-agent string

A user agent is the client name in the User-Agent HTTP header. Comscore gives this string:

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

The robots.txt token is proximic.

Block or allow it in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Comscore asks for a separate group for its crawler. If you want to keep it out of one section, add the path:

User-agent: proximic
Disallow: /path/

To block the whole site, use Disallow: /. To allow the crawler everywhere, use an empty Disallow: line, as in Comscore’s example:

User-agent: proximic
Disallow:

Comscore says that a robots.txt change can take up to a day to reach all its systems. Put the file at the top of your domain, for example www.example.com/robots.txt. A file in a subdirectory has no effect.

Limit the crawl rate

Comscore says that the crawler usually makes 1 request per second. If you want fewer requests, add Crawl-Delay. This group limits it to 1 request every 2 seconds:

User-agent: proximic
Crawl-Delay: 2

Verify requests

Comscore does not publish an IP list, and it says that the crawler’s IP addresses change from time to time. A block by IP address can stop working when the addresses change. If you need a secure way to allow the crawler, Comscore offers to add custom headers to its requests to your site. Contact Comscore for this.

What a block changes

A block asks the Comscore crawler to stay away. Comscore then cannot analyze those pages for its advertising partners. Comscore says that many premium publishers allow the crawler, because the analysis helps them sell and target ad inventory.