Sogou web spider: Sogou Search Crawler and robots.txt

Sogou web spider is the crawler of Sogou, a Chinese web search engine. It fetches pages so that people can find them in Sogou search. Sogou says that the spider supports robots.txt. If you want visitors from Sogou search, let it crawl. If you do not, block it in robots.txt.

What the spider does

Sogou explains the spider on its webmaster help page (in Chinese). The spider visits web pages, stores them in a local database and finds new links in them. This lets users find your pages in the Sogou search engine.

The crawler help page on the Sogou webmaster platform adds these facts:

1. The spider opens only one connection to a server with the same IP address.
2. It waits a few seconds between requests.
3. After it indexes a page, it waits at least a few days before it fetches that page again. Indexing is when a search engine stores a page for search.
4. It sets the refresh time of a page from the importance of the page and its change history.

User-agent string

A user agent is the client name in the User-Agent HTTP header. Sogou’s crawler help page gives this user agent for its PC crawler:

Sogou web spider

Sogou says that the user agent is case sensitive, so copy it exactly. The help page lists no user agents for other Sogou crawlers. Public bot lists, such as crawler-user-agents, record it with a version number:

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Block Sogou web spider in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Sogou says that its spider supports the robots protocol. If you want to keep it out of your whole site, add this group to /robots.txt:

User-agent: Sogou web spider
Disallow: /

If you want to block only one folder, use a path such as Disallow: /private/ instead.

The Sogou help page gives two warnings:

1. A change to robots.txt can take a few weeks to take effect.
2. Pages that you block cannot be found in Sogou search.

If the spider crawls too fast

Sogou asks you to contact it and send the spider entries from your access log. It asks site owners not to block the user agent for this reason. If the crawling never stops, Sogou asks you to look for pages that make new links on every visit.

Verify requests

Sogou does not publish an IP list for its spider. It gives a reverse DNS method instead. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log. On Linux or macOS, use host <ip>. On Windows, use nslookup <ip>.
2. Make sure that the hostname ends in .sogou.com.

Sogou says that a hostname outside *.sogou.com means that the request is fake.

Other Chinese search engines run their own crawlers, for example Baiduspider and 360Spider.