Intercom Fin: Website Sync Crawler and robots.txt

Fin is the AI agent from Intercom that answers customer questions in support conversations. It answers from content that the business adds, not from a live web search. A business can sync public website pages into Fin. Intercom says that this crawler has no dedicated user agent, and it ignores robots.txt unless the business turns on a setting.

Where Fin gets its answers

The Intercom article on the Fin AI Engine says that Fin searches the information, actions and data that it has access to. The content sources include help center articles, PDFs, approved URLs and past conversations. Intercom does not say that Fin searches the open web.

How website sync crawls pages

A business adds web pages to Fin with website sync. The business enters a URL, and the crawler fetches the pages under it. Intercom documents these facts about the crawler:

• It syncs public pages only. It cannot read content behind a login.
• It syncs up to 4,000 pages per source and up to 100 top-level domains.
• It can read the XML sitemap of the site to find more pages.
• It can load pages in a browser and run JavaScript when a page needs it.
• It re-syncs a source about once a week. Large or heavy sites move to a schedule of every 14 days.
• After the first 90 days, scheduled syncs pause when Fin did not use a page from the source in the last 90 days.

User agent and IP addresses

A user agent is the client name in the User-Agent HTTP header. Intercom says that website sync “does not use a dedicated, custom user-agent string at this time”. So you cannot find these requests in your logs by a Fin or Intercom name.

The crawler normally uses IP addresses that change. A customer can ask Intercom to turn on static, region-specific IP addresses for its workspace. The website sync article lists static proxy addresses for the United States, Europe and Australia.

robots.txt and website sync

Website sync has a “Respect robots.txt” setting, and it is off by default. When the business turns it on, the crawler obeys the disallow rules in your robots.txt file. Intercom does not name a robots.txt token for the crawler.

So robots.txt does not reliably keep your pages out of the Fin agent of another business. If you must stop it, block the requests by IP address or behavior on your web server, CDN or firewall.

Allow the crawler for your own Fin

If your own sync fails with a “Session blocked” or timeout error, Intercom says that bot protection or a firewall often causes it. In that case, ask Intercom for static IP addresses and allow them in your bot protection settings. Intercom adds that these requests are used only for website sync. They do not affect Messenger traffic.