ChatGLM-Spider: Zhipu AI Crawler and How to Block It

ChatGLM-Spider is a web crawler that bot directories link to Zhipu AI, the company behind the ChatGLM language models. We found no documentation for it from Zhipu AI: no user-agent page, no IP list and no statement about robots.txt. If you do not want it on your site, add a robots.txt rule and also block it on your server.

What is known

All facts below come from third-party bot directories that observe traffic. None of them comes from Zhipu AI.

Known Agents lists ChatGLM-Spider as an “AI Data Scraper” that Zhipu AI operates. It gives this user-agent string:

Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://zhipuai.cn/)

Centinel Analytica describes it as a crawler that collects data to train and evaluate Zhipu AI’s language models. It notes that the operator publishes no verification domains or IP ranges, so a match on the name alone does not prove the source.

What is not documented

We found no Zhipu AI page that documents these facts about ChatGLM-Spider:

1. The purpose of the crawler.
2. Whether the crawler obeys robots.txt.
3. The IP ranges or the reverse DNS host names of the crawler.

Without an IP list or reverse DNS, you cannot prove that a request comes from Zhipu AI. Anyone can send the string above.

Block ChatGLM-Spider in robots.txt

robots.txt is a text file at the site root with rules for crawlers. If you want to ask ChatGLM-Spider to stay away, add this group:

User-agent: ChatGLM-Spider
Disallow: /

RFC 9309, the robots.txt standard, says that these rules are not a form of access authorization. A crawler that does not obey robots.txt can still fetch your pages.

Block it on the server

If you want a block that does not depend on the crawler, block the user-agent token on your web server or firewall. This nginx example returns HTTP 403 for any user agent that contains the token:

if ($http_user_agent ~* "ChatGLM-Spider") {
    return 403;
}

Put the example in the server block of your site configuration, then reload nginx. A crawler that changes its user agent gets past this rule. If that happens, block the IP addresses from your logs instead.