# Zendesk AI: Web Crawler User Agent and Site Access

> Zendesk AI agents answer from help centers and crawled sites. The Zendesk/External-Content crawler and how to allow or block it.

Zendesk AI is the set of AI features in Zendesk, a customer service software company. Its AI agents and its generative search answer questions from content that a Zendesk customer connects, such as a help center or a crawled website. Zendesk gets external website content with a web crawler that sends the user agent `Zendesk/External-Content`.

## Where Zendesk AI gets its answers

Zendesk calls the content that an AI agent uses "knowledge sources". The [knowledge sources article](https://support.zendesk.com/hc/en-us/articles/8357749301658) lists two types:

- Zendesk help centers of the account or of a brand.
- External content that comes in through a web crawler or a knowledge connector.

An AI agent needs at least one knowledge source to create an AI-generated answer. For external sources, the agent searches the content from the last sync, which usually happens every 24 hours. If a customer is not signed in, the agent uses only public help center articles.

The Zendesk articles linked on this page describe only these knowledge sources. They do not describe a general web search for AI agents.

[Generative search](https://support.zendesk.com/hc/en-us/articles/8888178335898) shows an AI-generated answer when a user searches a help center. Zendesk says that the answer comes from the main search results, and users can open those articles for more information. If external content is connected, generative search can also use it.

## The Zendesk web crawler

A Zendesk admin sets up the crawler in Knowledge admin. The [web crawler article](https://support.zendesk.com/hc/en-us/articles/4593564000410) gives these facts:

- The crawler sends the user agent `Zendesk/External-Content`.
- It crawls a whole site, a sitemap, or a list of up to 500 URLs.
- It crawls only public pages that do not need a login.
- It follows links only inside the same domain.
- It can render pages in a browser when JavaScript adds content.
- It runs weekly by default. An admin can also set daily, monthly or a single sync.
- It does not follow `Crawl-delay` in robots.txt.
- It does not work with sites that use gzip compression.

Zendesk tells its customers to add sitemaps only for domains that they own, and to follow the terms of each website.

## Allow or block the crawler

If you want your Zendesk AI agent to use your website, allow the crawler at your firewall:

1. Add the user agent `Zendesk/External-Content` to the allow list of your firewall.
2. Allow the Zendesk IP ranges from the [Zendesk firewall article](https://support.zendesk.com/hc/en-us/articles/4408842860186).

If you do not want a Zendesk account to crawl your site, block the user agent `Zendesk/External-Content` in your web server or firewall. Zendesk does not say in these articles whether the crawler follows `Disallow` rules in robots.txt. It only says that the crawler ignores `Crawl-delay`. For this reason, do not depend on robots.txt alone.

A block on this crawler stops new crawls of your pages. Zendesk says that if a scheduled crawl fails, the help center keeps the results of the previous crawl.

---

[View the canonical page](https://aichatwatch.com/ai-chat-bot/zendesk-ai/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)