Zendesk AI: Web Crawler User Agent and Site Access

Zendesk AI is the set of AI features in Zendesk, a customer service software company. Its AI agents and its generative search answer questions from content that a Zendesk customer connects, such as a help center or a crawled website. Zendesk gets external website content with a web crawler that sends the user agent Zendesk/External-Content.

Where Zendesk AI gets its answers

Zendesk calls the content that an AI agent uses “knowledge sources”. The knowledge sources article lists two types:

• Zendesk help centers of the account or of a brand.
• External content that comes in through a web crawler or a knowledge connector.

An AI agent needs at least one knowledge source to create an AI-generated answer. For external sources, the agent searches the content from the last sync, which usually happens every 24 hours. If a customer is not signed in, the agent uses only public help center articles.

The Zendesk articles linked on this page describe only these knowledge sources. They do not describe a general web search for AI agents.

Generative search shows an AI-generated answer when a user searches a help center. Zendesk says that the answer comes from the main search results, and users can open those articles for more information. If external content is connected, generative search can also use it.

The Zendesk web crawler

A Zendesk admin sets up the crawler in Knowledge admin. The web crawler article gives these facts:

• The crawler sends the user agent Zendesk/External-Content.
• It crawls a whole site, a sitemap, or a list of up to 500 URLs.
• It crawls only public pages that do not need a login.
• It follows links only inside the same domain.
• It can render pages in a browser when JavaScript adds content.
• It runs weekly by default. An admin can also set daily, monthly or a single sync.
• It does not follow Crawl-delay in robots.txt.
• It does not work with sites that use gzip compression.

Zendesk tells its customers to add sitemaps only for domains that they own, and to follow the terms of each website.

Allow or block the crawler

If you want your Zendesk AI agent to use your website, allow the crawler at your firewall:

1. Add the user agent Zendesk/External-Content to the allow list of your firewall.
2. Allow the Zendesk IP ranges from the Zendesk firewall article.

If you do not want a Zendesk account to crawl your site, block the user agent Zendesk/External-Content in your web server or firewall. Zendesk does not say in these articles whether the crawler follows Disallow rules in robots.txt. It only says that the crawler ignores Crawl-delay. For this reason, do not depend on robots.txt alone.

A block on this crawler stops new crawls of your pages. Zendesk says that if a scheduled crawl fails, the help center keeps the results of the previous crawl.