Feedfetcher-Google: Google Feed Fetcher and Blocking

Feedfetcher is how Google fetches RSS and Atom feeds for Google News and WebSub. It acts for the users who add a feed, so it ignores robots.txt. If you want to stop it, your server must return an error status, such as 404 or 410, to the Feedfetcher-Google user agent.

User-agent string

Google’s list of user-triggered fetchers gives this string:

FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

Google’s Feedfetcher page writes the name as Feedfetcher-Google. If you match it in logs or server rules, ignore case.

What Feedfetcher does

Google lists Feedfetcher as a user-triggered fetcher. A fetcher usually makes a single request for a user. A crawler finds and scans sites automatically. Google’s Feedfetcher page gives these facts:

1. It fetches RSS and Atom feeds for Google News and WebSub. WebSub is a standard that tells subscribers when a feed changes.
2. It stores and refreshes feeds that users of an app or service requested.
3. It does not follow links. It fetches only the feed URL that users gave it.
4. It requests a common feed only once for all users who asked for it.
5. For most sites, it fetches a feed no more than once every hour on average. It can fetch sites that change often more frequently.

Google also says that only podcast feeds get indexed in Google Search.

robots.txt does not apply

Google says Feedfetcher acts as a direct agent of a human user, not as a robot. For this reason, it ignores robots.txt rules. A User-agent: Feedfetcher-Google group in your robots.txt has no effect.

Block Feedfetcher on the server

Google gives one method for this. If your feed is public, configure your server to return 404, 410 or another error status to the Feedfetcher-Google user agent. For example, in nginx, add this to the server block:

if ($http_user_agent ~* "feedfetcher-google") {
    return 410;
}

If a blog host or another hosting service serves your feed, ask that service to restrict access.

After the block, Google services that use Feedfetcher cannot get your feed. The rule matches only Feedfetcher, so Googlebot still crawls your web pages for Search.

Verify requests

Google publishes the Feedfetcher IP addresses in user-triggered-fetchers-google.json. Google’s verification guide says that IP addresses in this list resolve to a google.com hostname.

1. Run a reverse DNS lookup on the IP address from your log.
2. Make sure that the hostname ends in google.com.
3. Run a forward DNS lookup on that hostname.
4. Make sure that the result is the original IP address.

Related fetchers

Google Publisher Center uses a different fetcher, GoogleProducer, for feeds that publishers supply for Google News landing pages. Other feed readers run their own fetchers. For example, Feedly identifies itself as Feedly/1.0.