HubSpot Crawler: User Agent, IP Ranges and robots.txt
HubSpot Crawler fetches pages for HubSpot SEO tools, imports and the AI customer agent. Its user agent, IP ranges and robots.txt rules.
HubSpot Crawler is the bot that HubSpot uses to fetch web pages for its own tools. It reads pages for the SEO tools, for content imports and for the HubSpot AI customer agent. If you block it, these HubSpot features cannot read your pages.
What the crawler fetches
The HubSpot help center names three uses:
- SEO tools. HubSpot crawls pages, also external pages, to give SEO recommendations (SEO crawling errors).
- Content import. HubSpot crawls external pages that a user imports into HubSpot.
- Customer agent. This is an AI agent that answers customer questions from synced content (HubSpot).
For the customer agent, a HubSpot user can import public URLs. With the related URLs option on, HubSpot crawls up to 5,000 URLs on the domain. It syncs these sources again every week (knowledge sources). A HubSpot user starts each of these crawls. The account does not have to belong to the site owner.
User agent
HubSpot names its crawler HubSpot Crawler and asks site owners to allow this name in robots.txt. Cloudflare Radar lists three HubSpot bots as verified bots:
| Bot | User agent in Cloudflare Radar | Cloudflare category |
|---|---|---|
| HubSpot Crawler | HubSpot Crawler 1.0 http://www.hubspot.com/ |
Monitoring and analytics |
| HubSpot Page Fetcher | HubSpot Page Fetcher/1.0 http://www.hubspot.com/ [email protected] |
Page preview |
| HubSpot Feed Fetcher | HubSpot-FeedFetcher |
Feed fetcher |
Cloudflare says the two fetchers pull images through when a HubSpot user publishes a post to LinkedIn. For the bot that LinkedIn uses for link previews, see LinkedInBot.
robots.txt rules
HubSpot says its crawler follows the robots.txt standard (customer agent errors). If you want to block it from the whole site, add this group to /robots.txt:
User-agent: HubSpot Crawler
Disallow: /If you use HubSpot and its tools cannot crawl your pages, HubSpot asks you to add the crawler as an exemption:
User-agent: HubSpot Crawler
Allow: /HubSpot says that a noindex meta tag also stops the crawler from adding a page as a customer agent source. HubSpot also asks you to make sure that your robots.txt file is public and in the top-level folder of your site.
IP addresses
HubSpot publishes its IP ranges through a public IP ranges API. Ranges with the service WEB_SCRAPING have the description “Web crawling and user-generated traffic”. For the customer agent, HubSpot asks site owners to allow the range 54.174.58.224/27.
If you allow or block HubSpot in a firewall, use the list from the API. HubSpot says the API has the most up-to-date list. We found no reverse DNS method in the HubSpot documentation.
What blocking changes
If you block HubSpot Crawler, HubSpot SEO recommendations, page imports and customer agent sources that use your pages stop working. The rule names only HubSpot Crawler. It does not change how search engines or AI crawlers treat your site. For those bots, see AI crawlers on your site.