# Glean: Enterprise AI Search and Its Website Crawler

> Glean searches a company's own apps, not the public web. How its Website connector crawls sites and follows robots.txt, and how its web search works.

Glean is an enterprise search and AI assistant product. It searches the apps and documents of one company for the employees of that company. It is not a public web search engine. A public website can reach Glean users in two ways. A Glean customer can crawl the site with the Website connector. Or Glean web search can find its pages through OpenAI, Brave or Google.

## What Glean is

Glean connects to workplace apps through connectors. A connector links Glean to one app and indexes its content. Examples are Slack, Salesforce, Zendesk and Zoom. Glean's documentation lists native connectors for many apps. One of them, the Website connector, crawls websites. Glean says that the data of each customer is isolated in its own tenant.

## The Website connector

The [Website connector](https://docs.glean.com/connectors/native/website/) is a crawler that a Glean admin at the customer company sets up for one site. These are the documented facts that matter to a site owner:

- Glean finds pages from a sitemap or from seed URLs, and then follows links.
- A full crawl runs every 28 days by default.
- The connector honors robots.txt by default. An admin can turn off "Honor robots.txt". Glean's documentation says to do this only when the site owner approves.
- Glean does not publish the default user-agent string. An admin can set a custom user agent.
- Requests come from the proxy egress IP addresses of the customer's Glean instance. Glean does not publish one list. Each customer gets its addresses from Glean Support.
- The connector can render JavaScript pages in headless Chrome, but only for public pages.
- It can log in with credentials that the site gives, for example basic authentication, bearer tokens or cookies.

Pages that the connector indexes are visible to all users of that connector in the customer's Glean. The connector does not copy page-level permissions from the website.

## Block or allow the Website connector

Glean documents no robots.txt token for its crawler. Under the robots.txt standard ([RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)), a crawler without its own group follows the `User-agent: *` group. If you want to keep a section away from all crawlers that follow robots.txt, disallow it in that group:

```
User-agent: *
Disallow: /private/
```

This rule also applies to Googlebot and to every other crawler that has no group of its own in your file. A Glean admin can turn off "Honor robots.txt". If you must stop crawling fully, block the requests on your server or firewall. You need the user agent and IP addresses from your own logs, or from the company that runs the crawl.

## Web search in Glean Assistant

Glean Assistant and Glean Agents can also search the public web. Admins add a web search tool in the admin console. The [web search setup guide](https://docs.glean.com/administration/tools/setup-tools/web-search-tools-setup.md) lists three providers: OpenAI, Brave and Google Gemini.

For this feature, the search runs through the provider that the admin picks. Whether your pages appear depends on the index and the controls of that provider. See [OAI-SearchBot](/ai-crawler-bot/oai-searchbot/) for OpenAI search and [Googlebot](/ai-crawler-bot/googlebot/) for Google. Google's [Google-Extended](/ai-crawler-bot/google-extended/) token covers grounding in Grounding with Google Search on Vertex AI. Glean's documentation links to that Google option.

---

[View the canonical page](https://aichatwatch.com/ai-search-engine/glean/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)