MLBot: Undocumented Crawler User Agent and Blocking
MLBot has no operator documentation. What public bot lists say, what is unknown, and how to block it with robots.txt and nginx.
Table of Contents
MLBot is a user agent that no operator documents today. Public bot lists connect it to a company called Metadata Labs, but that bot page no longer exists. If you see MLBot in your logs and do not want it, block it in robots.txt and at your server.
What is known about MLBot
The open crawler-user-agents list on GitHub records this user-agent string:
MLBot (www.metadatalabs.com/mlbot)The list describes MLBot as a Metadata Labs web crawler for analysis. On 2026-10-22, the address www.metadatalabs.com/mlbot sent visitors to a parked landing page with no information about a bot.
What is not known
We found no operator page that states any of these facts:
- What MLBot collects, or why.
- Whether MLBot follows robots.txt.
- Which IP addresses or reverse DNS names MLBot uses.
The letters “ML” in the name do not prove a link to machine learning. We found no source that says MLBot collects data to train AI models.
Block MLBot
To ask MLBot to stay away, add this group to your robots.txt file:
User-agent: MLBot
Disallow: /No source confirms that MLBot reads robots.txt. If the requests continue after you add the group, block the user agent at your web server, CDN or firewall. This nginx example returns HTTP 403 to requests whose user agent starts with MLBot:
if ($http_user_agent ~ "^MLBot") {
return 403;
}Put the rule in the server block of the nginx configuration for your site. The match is case-sensitive and anchored to the start, so it does not catch longer names that contain the same letters.
A user-agent rule stops only clients that send this string. If the same traffic comes back with a different user agent, block its IP addresses instead.
For crawlers that publish their purpose and their robots.txt behavior, see the introduction to AI crawler bots.