# Coveo: Enterprise Search Software and Coveobot Crawler

> Coveo is search software that companies run on their own sites. How its crawler Coveobot handles robots.txt, noindex and crawl rate.

Coveo is commercial search software. A company uses it to add search and AI-generated answers to its own website, support portal or online store. It is not a public AI search engine, so you cannot submit your site to it. You meet Coveo in two ways. You run it on your own site, or its crawler Coveobot visits your site for a Coveo customer.

## How Coveo indexes a website

A Coveo customer connects content to Coveo through sources. For a website, the customer adds a [Web source](https://docs.coveo.com/en/malf0160/). Coveo says that the Web source works like the bots of web search engines. It starts at one URL, follows the links on each page, and indexes the pages it finds. By default it rescans the site every day.

Coveo can crawl a public website without proof that the customer owns it. Proof of ownership is needed only for a crawl rate faster than one page per second.

## Coveobot user agent

The default user agent of the Web source crawler is:

```
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html)
```

The default name that the crawler looks for in robots.txt is `CoveoBot` (the [`RobotsDotTextUserAgentString`](https://docs.coveo.com/en/mc1f0219/) setting). The Coveo customer can change both values.

## robots.txt, noindex and nofollow

By default the crawler obeys every directive it finds. These are robots.txt rules, `noindex` and `nofollow` in a robots meta tag or an `X-Robots-Tag` header, and `rel="nofollow"` on links. The customer can turn on an override for each of these directives. Coveo's [crawler directives page](https://docs.coveo.com/en/mc2a1538/) calls an override a last resort and recommends rules for `coveobot` instead.

If you do not want Coveo customers to crawl your site, add this group to robots.txt:

```
User-agent: CoveoBot
Disallow: /
```

This rule works only while the customer keeps the default robots.txt name and does not turn on the robots.txt override.

If you want Coveo to index a page that other crawlers must not index, give Coveo its own meta tag:

```html

```

When a page has rules for `coveobot`, the crawler follows them and ignores the general robots rules. The [meta robots guide](/html-tags/meta-robots/) explains the tag.

## Crawl rate

The crawler requests at most one page per second by default. A faster rate needs proof of ownership: a file named `coveo-ownership-orgid.txt` at the site root, where `orgid` is the Coveo organization ID. If robots.txt sets a `Crawl-delay` with a different value, the slower rate applies.

## IP addresses

Coveo publishes the [IP addresses](https://docs.coveo.com/en/1831/) that its cloud crawlers use, for each region. A customer can also run the Coveo Crawling Module on its own servers, and that module can crawl internal websites. So not every Coveobot request comes from the listed addresses.

## Generated answers

Coveo sells Relevance Generative Answering (RGA) as a paid add-on. [RGA](https://docs.coveo.com/en/n9de0370/) writes an answer only from the content in the customer's Coveo index. The answer cites the indexed items that it used. By default RGA works only in English.

So a Coveo answer on a company site uses that company's index. Your pages are in it only if that customer indexed them.

---

[View the canonical page](https://aichatwatch.com/ai-search-engine/coveo/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)