Feedly Fetcher (FeedlyBot): User Agent and Blocking

Feedly Fetcher, called FeedlyBot in some bot lists, gets the RSS and Atom feeds that Feedly users follow. It ignores robots.txt. If you want to stop it, your server must return an error status such as 404 or 410 to the Feedly/1.0 user agent.

Name and user agent

Feedly describes this program on two pages with the same text: Feedly Fetcher and Feedly Poller. Neither page uses the name FeedlyBot.

Both pages identify the program by this user agent:

Feedly/1.0

Feedly does not publish the full string. Real requests can add more text after Feedly/1.0. For example, the logs in this post by Seth Larson show a subscriber count in the string. If you match Feedly in logs or server rules, match the Feedly/1.0 prefix.

What Feedly Fetcher does

Feedly’s pages give these facts:

1. It gets a feed when a user adds the feed to Feedly or to an app built on the Feedly cloud.
2. It refreshes these feeds from time to time.
3. It does not follow links. It fetches only the feed URLs that users added.
4. It requests a common feed only once for all users who follow it.
5. For most sites, it fetches a feed no more than once every hour on average. It can fetch sites that change often more frequently.
6. If your feed names a push hub, Feedly subscribes to the hub and polls only once a day.

A push hub is a server that sends feed updates to subscribers. WebSub is the standard for this.

robots.txt does not apply

Feedly says the fetcher acts as a direct agent of the human user, not as a robot. For this reason, it ignores robots.txt rules. A robots.txt group for Feedly has no effect.

Block Feedly on the server

Feedly’s method is an error status. If your feed is public, configure your server to return 404, 410 or another error status to the Feedly/1.0 user agent. For example, in nginx, add this to the server block:

if ($http_user_agent ~* "^Feedly/") {
    return 410;
}

If a blog host or another hosting service serves your feed, ask that service to restrict access. After the block, Feedly cannot fetch your feed for its users.

Verify requests

Feedly does not publish its IP addresses. Its pages say the addresses change from time to time, and that the user agent is the best way to identify the fetcher. Feedly does not document a reverse DNS method. So you cannot prove that a request with this user agent comes from Feedly.

Google runs a similar program for its products, Feedfetcher. It also ignores robots.txt.