Cloudflare Always Online Crawler: User Agents and IPs
Cloudflare Always Online shows archived pages when your origin is down. The user agents and IP ranges that crawl your site, and how to turn it off.
Always Online is a Cloudflare feature that shows visitors an archived copy of your pages when Cloudflare cannot reach your origin server. Cloudflare sends your popular URLs to the Internet Archive, and the Internet Archive’s crawler saves them. It is not an AI crawler. If you do not want this crawl, turn Always Online off in the Cloudflare dashboard.
How Always Online works
In September 2020, Cloudflare announced a partnership with the Internet Archive for Always Online. Cloudflare’s Always Online documentation describes the current process:
- Cloudflare shares your hostname and popular URL paths with the Internet Archive.
- Cloudflare picks the most popular URLs among GET requests that returned HTTP 200 in the previous five hours.
- The Internet Archive’s crawler saves those pages in the Wayback Machine.
- If Cloudflare cannot reach your origin, it first looks for a stale copy in its own cache.
- If no cached copy exists, Cloudflare serves the static parts of the page from the Internet Archive, with a banner for the visitor.
Cloudflare limits how often the crawl runs:
| Cloudflare plan | Crawl interval |
|---|---|
| Free | Every 30 days |
| Pro | Every 15 days |
| Business | Every 5 days |
| Enterprise | Every 5 days |
Always Online starts only when Cloudflare cannot connect to the origin, which gives Cloudflare status codes 520 to 527. It does not start when your origin answers with a 404, 500 or 503 error. It cannot show pages behind a login, and it cannot handle form submissions.
User agents
Cloudflare documents two user-agent strings for this feature. The user agent is the client name in the User-Agent HTTP header. The Cloudflare crawlers reference lists this string for Always Online:
Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online)The Always Online troubleshooting page says that the Internet Archive currently uses this string when it crawls for Always Online:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/605.1.15The second string looks like a normal browser. You cannot identify these requests by the user agent alone, so use the IP ranges below.
IP ranges
Per Cloudflare, the crawl comes from these Internet Archive ranges, not from Cloudflare addresses:
207.241.224.0/20
208.70.24.0/21Turn Always Online off
Cloudflare’s documentation does not give a robots.txt token for Always Online. The direct control is the setting in the dashboard:
- In the Cloudflare dashboard, open the Caching Configuration page.
- Select the domain.
- Under Always Online, set the toggle to Off.
When Always Online is off, Cloudflare stops sending your URLs to the Internet Archive for this feature. The Internet Archive can still crawl your site for the Wayback Machine in other ways. See the archive.org_bot guide.
Keep Always Online working
If you use Always Online, Cloudflare gives these rules:
- Do not block the Internet Archive IP ranges above.
- Do not block Known Bots or Verified Bots with a WAF custom rule.
- Do not block the United States with an IP Access rule or a WAF custom rule.
- Do not use Always Online for API traffic.
To see if a page was saved, search for its URL in the Wayback Machine. Cloudflare sends only popular pages. If you want to archive a specific page, submit it at web.archive.org/save.