ImagesiftBot: ImageSift Crawler User Agent and robots.txt

ImagesiftBot is the crawler of ImageSift, which its about page calls “Imagesift by Hive”. It downloads public images and page text for an image search index. ImageSift says that the bot follows robots.txt rules, so you can block it with a robots.txt group.

What it collects

The ImageSift about page says the crawler collects public images, the host URL, the text on the page and the alt text of the images. ImageSift stores this data in an index. Its web intelligence products use the index to search for and retrieve similar images.

ImageSift does not say that it uses the data to train AI models. Some AI crawler block lists, for example ai.robots.txt, include ImagesiftBot.

User agent

The ImageSift about page gives this user agent:

Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)

The robots.txt token is ImagesiftBot.

Block ImagesiftBot in robots.txt

ImageSift says that the bot respects standard robots.txt rules that name ImagesiftBot. If you want to block the whole site, add this group to /robots.txt:

User-agent: ImagesiftBot
Disallow: /

If you want to block one folder only, use the example from the ImageSift page:

User-Agent: ImagesiftBot
Allow: /
Disallow: /private/

Googlebot fallback and Crawl-delay

The ImageSift page gives two more rules:

• If no group names ImagesiftBot but a group names Googlebot, ImagesiftBot follows the Googlebot group.
• ImagesiftBot supports Crawl-delay. The value is the minimum number of seconds between the start of two requests.

The first rule matters if your file has a Googlebot group. If your file allows Googlebot and blocks all other bots with User-agent: *, ImagesiftBot follows the Googlebot group and crawls your site. If you want a different result, add a group for ImagesiftBot.

Opt out and identify requests

If you want ImageSift to stop crawling your site, you can also send an email to [email protected]. The about page gives this address for opt-out requests.

ImageSift publishes no IP list and no reverse DNS method. Any client can send the ImagesiftBot user agent. If you must stop all requests with this name, block the user agent on your server or in your firewall. For more on finding bots in your logs, see AI crawlers on your site.