Salesforce Einstein and Agentforce: Web Search and Crawlers
Salesforce Einstein is the AI that Salesforce builds into its platform. Its conversational assistant is now called Agentforce. Agentforce agents answer mostly from a company’s own data, but since 2025 they can also use web search and website connectors. The Salesforce announcements do not name a user agent for this web access.
Einstein and Agentforce
Salesforce says that Einstein is built into the Salesforce Platform. It uses data from the CRM and from external applications to make predictions and to generate content (Salesforce). CRM (customer relationship management) software keeps records about customers and sales.
The same Salesforce page gives Agentforce as the new name of Einstein Copilot. An editor’s note on a Salesforce newsroom article also says that the assistant was upgraded to Agentforce.
Web content in Agentforce
The Summer ’25 release announcement from June 16, 2025 lists three features that involve websites:
| Feature | What Salesforce says it does |
|---|---|
| Web Search in Agentforce Data Library | Lets agents use information sources on the public web |
| Web Crawler and Site Map connectors | Bring content from a company’s website and other external repositories into Agentforce |
| In-line citations | Show the data sources of a response, as part of the RAG tools |
RAG (retrieval-augmented generation) is a method in which the model first finds documents and then answers from them.
In June 2026, Salesforce announced Agentforce Help Agent, a service agent for customer support. Salesforce says that its users can add files or enter a web URL for crawling (Salesforce newsroom).
User agents and blocking
The announcements above do not name a user agent, a robots.txt token or IP ranges for web search or for the crawler connectors. We did not find a public Salesforce page that documents them. For this reason, this guide cannot give a robots.txt rule for Agentforce.
If you want to control this access, use your server logs:
- Find the requests from crawlers that you do not recognize.
- Write down the user agent and the IP address of each request.
- If you do not want these requests, block that user agent or those IP addresses in your web server or firewall.
The guide How to Find and Block AI Crawlers shows these steps in more detail. Some vendors document the crawler of their support AI. For example, Zendesk names the user agent of its knowledge crawler (Zendesk AI guide).