Pinterestbot: Pinterest Crawler User Agent and robots.txt

Pinterestbot is the crawler of Pinterest. It indexes public pages so that people can find them on Pinterest, and it keeps Pin details such as price and title up to date. Pinterest says that Pinterestbot obeys robots.txt and that it does not use the crawled content to train its image model, Pinterest Canvas.

What Pinterestbot does

Pinterest’s Pinterestbot help page names two functions:

Function	What Pinterest says it does
Crawling	Indexes public pages. For products it collects the image, title, description, price and availability. For articles it shows only a preview on Pins.
Scraping	Reads the pages behind Pins to keep Pin details current, and finds and removes broken links behind Pins

Pinterest states: “We do not use content crawled by Pinterestbot to train our image generation foundation model, Pinterest Canvas.”

User-agent strings

A user agent is the client name in the User-Agent HTTP header. Pinterest lists these three strings:

Pinterest/0.2 (+https://www.pinterest.com/bot.html)

Mozilla/5.0 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)

The robots.txt token is Pinterestbot.

Block or limit Pinterestbot in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Pinterest asks you to put the file on your main domain, because it does not support robots.txt files on subdomains. If you want to block Pinterestbot on the whole site, add this group:

user-agent: Pinterestbot
disallow: /

If you want to block one folder but allow a subfolder in it, use this example from Pinterest:

user-agent: Pinterestbot
disallow: /directory1/
allow: /directory1/subdirectory1/

Pinterest also reads crawl-delay. It honors values up to 1 and treats larger values as 1. Pinterest says that large crawl delays reduce the distribution and recommendation of your content on Pinterest.

What a block changes

A block asks Pinterestbot to stop crawling your site. Pinterest uses these crawls to index your pages and to keep the details of Pins current.

Pinterest documents a different control for saves. If you do not want people to save things from your site to Pinterest, use its nopin meta tag. Put it in the <head> of each page:

<meta name="pinterest" content="nopin" />

To turn off saving for one image, add nopin="nopin" to its <img> tag.

Verify requests

Pinterest says that it crawls mainly from US addresses, which are always in the range 54.236.1.XXX. Crawls from outside the US have no fixed range. Pinterest tells you not to hard code these addresses, and to use reverse DNS instead. Do these steps on a host with the host command:

1. Run host <IP address> on the IP address from your logs.
2. Make sure that the hostname ends in pinterest.com or pinterestcrawler.com.
3. Run host on that hostname.
4. Make sure that the result is the same IP address as in your logs.

If a client sends a Pinterest user agent but fails this test, Pinterest asks you to contact it.