IBM watsonx Assistant: Web Crawler and Search Sources

Move to watsonx Orchestrate
Where answers come from
The Watson Discovery web crawler
Block or allow the crawler

IBM watsonx Assistant is a platform that businesses use to build chatbots for their customers. It does not search the open web. It answers from content that the business connects, for example a Watson Discovery collection made from a web crawl. On IBM Cloud, that crawler sends the user agent IBM-AppConnect/V1.

Move to watsonx Orchestrate

IBM is moving watsonx Assistant customers to watsonx Orchestrate. The watsonx Assistant release notes for 20 September 2026 describe automatic upgrades to watsonx Orchestrate. IBM is rolling them out for some Standard, Plus and Enterprise plans. Existing actions, dialogs and extensions keep working after the upgrade.

The same note says that IBM now limits the creation of new watsonx Assistant instances. IBM tells customers to create a watsonx Orchestrate instance for assistant features. The old product page on ibm.com now redirects to watsonx Orchestrate.

Where answers come from

Conversational search sends search results to an IBM watsonx model, which writes the reply. The results come from a search service that the business connects. IBM lists IBM Watson Discovery, Elasticsearch, custom search and Milvus.

The assistant can show citations. If the builder sets a citation title, the reply includes a list of references to the source content that the assistant used.

The Watson Discovery web crawler

Watson Discovery can build a collection from a website with a web crawl data source. The IBM Cloud web crawl documentation gives the identifying header of the crawler:

User-Agent: IBM-AppConnect/V1

A user agent is the client name in the User-Agent HTTP header. IBM tells its customers to add this crawler to the allowlist of a site that returns HTTP 403 to it. The IBM Cloud page does not say whether the crawler reads robots.txt.

The self-managed version of Discovery for Cloud Pak for Data has an “Ignore robots.txt” setting. When a customer turns it on, the crawler ignores the allow and deny rules in robots.txt (Cloud Pak for Data web crawl). IBM does not give a user agent on that page.

If a business uses Elasticsearch or a custom search service, that service fetches the pages. The watsonx Assistant documentation does not name a user agent for those services.

Block or allow the crawler

If your own assistant uses pages from your site and the crawl fails with HTTP 403, allow the IBM-AppConnect/V1 user agent. Add the rule on your web server, CDN or firewall.

If you do not want Watson Discovery collections to copy your pages, block that user agent on your server. A robots.txt rule does not stop a crawler that its operator set to ignore robots.txt. Any client can send this user-agent string, and IBM publishes no IP list to confirm real requests.