Coveo: Enterprise Search Software and Coveobot Crawler

Coveo is commercial search software. A company uses it to add search and AI-generated answers to its own website, support portal or online store. It is not a public AI search engine, so you cannot submit your site to it. You meet Coveo in two ways. You run it on your own site, or its crawler Coveobot visits your site for a Coveo customer.

How Coveo indexes a website

A Coveo customer connects content to Coveo through sources. For a website, the customer adds a Web source. Coveo says that the Web source works like the bots of web search engines. It starts at one URL, follows the links on each page, and indexes the pages it finds. By default it rescans the site every day.

Coveo can crawl a public website without proof that the customer owns it. Proof of ownership is needed only for a crawl rate faster than one page per second.

Coveobot user agent

The default user agent of the Web source crawler is:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html)

The default name that the crawler looks for in robots.txt is CoveoBot (the RobotsDotTextUserAgentString setting). The Coveo customer can change both values.

robots.txt, noindex and nofollow

By default the crawler obeys every directive it finds. These are robots.txt rules, noindex and nofollow in a robots meta tag or an X-Robots-Tag header, and rel="nofollow" on links. The customer can turn on an override for each of these directives. Coveo’s crawler directives page calls an override a last resort and recommends rules for coveobot instead.

If you do not want Coveo customers to crawl your site, add this group to robots.txt:

User-agent: CoveoBot
Disallow: /

This rule works only while the customer keeps the default robots.txt name and does not turn on the robots.txt override.

If you want Coveo to index a page that other crawlers must not index, give Coveo its own meta tag:

<meta name="robots" content="nofollow, noindex" />
<meta name="coveobot" content="all" />

When a page has rules for coveobot, the crawler follows them and ignores the general robots rules. The meta robots guide explains the tag.

Crawl rate

The crawler requests at most one page per second by default. A faster rate needs proof of ownership: a file named coveo-ownership-orgid.txt at the site root, where orgid is the Coveo organization ID. If robots.txt sets a Crawl-delay with a different value, the slower rate applies.

IP addresses

Coveo publishes the IP addresses that its cloud crawlers use, for each region. A customer can also run the Coveo Crawling Module on its own servers, and that module can crawl internal websites. So not every Coveobot request comes from the listed addresses.

Generated answers

Coveo sells Relevance Generative Answering (RGA) as a paid add-on. RGA writes an answer only from the content in the customer’s Coveo index. The answer cites the indexed items that it used. By default RGA works only in English.

So a Coveo answer on a company site uses that company’s index. Your pages are in it only if that customer indexed them.