# Sogou web spider: Sogou Search Crawler and robots.txt

> Sogou web spider crawls pages for Sogou, a Chinese search engine. Its user agent, robots.txt support, reverse DNS check and how to block it.

Sogou web spider is the crawler of Sogou, a Chinese web search engine. It fetches pages so that people can find them in Sogou search. Sogou says that the spider supports robots.txt. If you want visitors from Sogou search, let it crawl. If you do not, block it in robots.txt.

## What the spider does

Sogou explains the spider on its [webmaster help page](https://www.sogou.com/docs/help/webmasters.htm) (in Chinese). The spider visits web pages, stores them in a local database and finds new links in them. This lets users find your pages in the Sogou search engine.

The [crawler help page](https://zhanzhang.sogou.com/index.php/help/spider) on the Sogou webmaster platform adds these facts:

1. The spider opens only one connection to a server with the same IP address.
2. It waits a few seconds between requests.
3. After it indexes a page, it waits at least a few days before it fetches that page again. Indexing is when a search engine stores a page for search.
4. It sets the refresh time of a page from the importance of the page and its change history.

## User-agent string

A user agent is the client name in the User-Agent HTTP header. Sogou's crawler help page gives this user agent for its PC crawler:

```text
Sogou web spider
```

Sogou says that the user agent is case sensitive, so copy it exactly. The help page lists no user agents for other Sogou crawlers. Public bot lists, such as [crawler-user-agents](https://github.com/monperrus/crawler-user-agents), record it with a version number:

```text
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
```

## Block Sogou web spider in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Sogou says that its spider supports the robots protocol. If you want to keep it out of your whole site, add this group to `/robots.txt`:

```text
User-agent: Sogou web spider
Disallow: /
```

If you want to block only one folder, use a path such as `Disallow: /private/` instead.

The Sogou help page gives two warnings:

1. A change to robots.txt can take a few weeks to take effect.
2. Pages that you block cannot be found in Sogou search.

## If the spider crawls too fast

Sogou asks you to contact it and send the spider entries from your access log. It asks site owners not to block the user agent for this reason. If the crawling never stops, Sogou asks you to look for pages that make new links on every visit.

## Verify requests

Sogou does not publish an IP list for its spider. It gives a reverse DNS method instead. A reverse DNS lookup finds the hostname for an IP address.

1. Run a reverse DNS lookup on the IP address from your log. On Linux or macOS, use `host `. On Windows, use `nslookup `.
2. Make sure that the hostname ends in `.sogou.com`.

Sogou says that a hostname outside `*.sogou.com` means that the request is fake.

Other Chinese search engines run their own crawlers, for example [Baiduspider](/ai-crawler-bot/baiduspider/) and [360Spider](/ai-crawler-bot/360spider/).

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/sogou-spider/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)