Diffbot Crawler: User Agents, robots.txt and Blocking
Diffbot's two user agents, Diffbot and Diffbot-User, what each one does, how Diffbot handles robots.txt and crawl-delay, and how to block them.
Diffbot is a company that extracts structured data from web pages and builds a Knowledge Graph from it. Its own crawler uses the token Diffbot, and requests that a person starts through Diffbot software use Diffbot-User. Diffbot says that its crawls follow robots.txt by default, and this includes crawl-delay. If you want to block or slow Diffbot, add a robots.txt group for each token.
Diffbot user agents
Diffbot’s robots.txt FAQ lists two user agents:
| Token | Purpose, as Diffbot describes it |
|---|---|
Diffbot |
General web crawling to build a search engine. Sites can then be found and cited in the Diffbot Knowledge Graph and Diffbot web search. Diffbot says that it is not used for AI training. |
Diffbot-User |
Requests on behalf of a person who opens a URL with Diffbot software. |
The FAQ also states that Diffbot does not crawl pages to train generative AI foundation models. The FAQ gives no full user agent strings, no IP ranges and no reverse DNS host name.
Crawls that customers run
Diffbot customers can run their own crawls with the Crawlbot and Extract APIs. The customer sets the crawl parameters. Diffbot recommends that customers set a user agent that names their own organization. So a crawl that runs on Diffbot can show a different name in your logs.
The Crawlbot documentation says that Crawlbot follows robots.txt by default, and this includes crawl-delay. A customer can turn this off. Diffbot calls this a rare case, for example a partner site that gave permission. Crawlbot also waits 0.25 seconds between page loads by default, and the customer can change this delay.
Block Diffbot in robots.txt
robots.txt is a text file at the root of a site that gives rules to crawlers (RFC 9309). Diffbot lists its two tokens separately. If you want to block both, add one group for each:
User-agent: Diffbot
Disallow: /
User-agent: Diffbot-User
Disallow: /If you want to slow the crawler, set a crawl delay. Crawl-delay is not part of RFC 9309, but Diffbot says that it follows it:
User-agent: Diffbot
Crawl-delay: 10Server and firewall blocks
Diffbot publishes no IP ranges. You can refuse requests that contain Diffbot in the user agent on your web server. Any client can send that name, so the user agent alone does not prove that Diffbot sent a request. A server block by name also misses customer crawls that use their own user agent.
What a block changes
A block on Diffbot keeps your pages out of the Diffbot crawl that feeds its Knowledge Graph and web search. Diffbot says that this crawl is not used for AI training, so the block has no documented effect on training data. A rule for Diffbot applies only to Diffbot. It does not change how Googlebot or Bingbot treat your site.