Webzio-extended and Omgilibot: Webz.io Bots and robots.txt

Webzio-extended is one of two bots that Webz.io introduced in 2024 after Omgilibot. Webz.io says that it tags collected data as usable or not usable for AI and machine learning training. Webz.io publishes no user-agent string and no robots.txt token for it. If you do not want Webz.io to collect your pages, block all of its crawler names in robots.txt and on your server.

What Webz.io documents

Webz.io sells web data through APIs. On July 17, 2024, it published the post From Omgilibot to the Webzio Duo. The address webz.io/bot.html now leads to this post. The post describes two bots:

Bot	What Webz.io says it does
Webzio	Crawls websites and structures the content
Webzio-extended	Takes the data that Webzio collected and tags it as usable or not usable for AI and machine learning training

The post says that both bots obey robots.txt exclusions. It also says that they find content that is marked as off limits for AI use. For AI purposes, Webz.io says that it supplies only data that is tagged as explicitly permitted.

What the post does not say

The post leaves out these facts:

• The full user-agent string of each bot
• The exact robots.txt token of each bot
• The signal that marks content as off limits for AI use
• Whether Webzio-extended sends its own requests or only tags data that Webzio fetched

A user agent is the client name in the User-Agent HTTP header. Public bot lists, such as crawler-user-agents, record the Webzio crawler with this user agent:

webzio (+https://webz.io/bot.html)

That list has no separate entry for Webzio-extended.

Omgilibot, the older Webz.io crawler

Webz.io explained Omgilibot in a 2017 post. It built the bot for the Omgili search engine, which it later discontinued. The bot then collected pages for the Webz.io API, with a focus on news sites, blogs, reviews, ecommerce and discussion forums. The post gives this user agent:

omgili/0.5 +https://omgili.com

Third-party bot lists use the name omgilibot. The post also says that Webz.io always complies when a site owner asks it to stop, through robots.txt or directly.

Block Webz.io crawlers in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Because Webz.io does not document the tokens, list the names from the post and the older Omgilibot names in one group:

User-agent: Webzio
User-agent: Webzio-extended
User-agent: omgili
User-agent: omgilibot
Disallow: /

Case does not matter. Under RFC 9309, crawlers match these names without regard to case.

If you want to keep your content out of AI training data only, add a group for Webzio-extended alone. This name matches the role that the post gives that bot:

User-agent: Webzio-extended
Disallow: /

Webz.io does not confirm that this group has an effect.

Block on the server

If you want to make sure that the requests stop, block them on your web server too. This nginx example returns HTTP 403 for any user agent that contains webzio or omgili:

if ($http_user_agent ~* "(webzio|omgili)") {
    return 403;
}

Put the example in the server block of your site configuration, then reload nginx.

Verify requests

Webz.io publishes no IP list and no reverse DNS method for its crawlers. Anyone can copy a user-agent string. A match in your logs does not prove that the request came from Webz.io.