Bytespider: ByteDance Crawler User Agent and Blocking
Bytespider is the ByteDance crawler for Toutiao Search, reported as an AI training crawler. Its documented user agent, 2019 IP list and how to block it.
Bytespider is a web crawler of ByteDance, the company behind TikTok. ByteDance’s own documentation names it as the crawler of Toutiao Search. In 2024, Cloudflare wrote that Bytespider is reportedly used to collect training data for ByteDance’s large language models (LLMs). ByteDance’s documentation does not say whether Bytespider follows robots.txt. If you do not want it on your site, add a robots.txt rule and also block it on your server or CDN.
What ByteDance documents
The only ByteDance page about Bytespider that we found is a short post from the Toutiao Search account, 头条搜索spider介绍 (“Toutiao Search spider introduction”). It is in Chinese and dates from November 2019. The post says these things:
- The Toutiao Search crawler uses the user agent token
Bytespider, with a capital B. - The crawler came from six IP ranges at that time (listed below).
- If Bytespider crawls too much and slows down your site, set a lower crawl rate. Use the crawl frequency tool on the Toutiao Search webmaster platform (zhanzhang.toutiao.com). The setting takes effect within one day.
- For other questions about the crawler, write to [email protected].
The post does not mention robots.txt or AI training.
User agent string
This is the example from the 2019 post:
Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML,like Gecko)Chrome/41.0.6633.1032 Mobile Safari/537.36;Bytespider;https://zhanzhang.toutiao.comTo find the crawler in your access logs, search for the token Bytespider.
IP ranges from 2019
The 2019 post lists these ranges:
110.249.201.0/24
110.249.202.0/24
111.225.148.0/24
111.225.149.0/24
220.243.135.0/24
220.243.136.0/24The list on that page is from 2019, and ByteDance does not document a DNS verification method. Treat the list as a hint, not as a complete or current list.
AI training use
In July 2024, Cloudflare reported that Bytespider made the most requests of all AI bots on its network. Cloudflare wrote that Bytespider is reportedly used to gather training data for ByteDance’s LLMs, including the models behind its Doubao assistant. ByteDance does not state this purpose on the page above.
Block Bytespider
If you want to block it with robots.txt, add this group:
User-agent: Bytespider
Disallow: /Because ByteDance does not document how Bytespider treats robots.txt, add a server rule too. This example for nginx returns 403 to any request whose user agent contains Bytespider:
if ($http_user_agent ~* "Bytespider") {
return 403;
}If your site is on Cloudflare, you can also turn on its managed block for AI crawlers, which the same Cloudflare post introduced.
Related ByteDance bots
Bot lists connect other tokens with ByteDance, for example TikTokSpider and ByteDance-Frontpage. The ByteDance post above does not mention them. ByteDance publishes no documentation for ByteDance-Frontpage, so its purpose and its robots.txt behavior are not known. A robots.txt group for Bytespider does not apply to other tokens, so name each bot that you want to block:
User-agent: ByteDance-Frontpage
Disallow: /