# Webzio-extended and Omgilibot: Webz.io Bots and robots.txt

> Webzio-extended tags Webz.io crawl data as usable or not for AI training. What Webz.io documents about it and Omgilibot, and how to block its bots.

Webzio-extended is one of two bots that Webz.io introduced in 2024 after Omgilibot. Webz.io says that it tags collected data as usable or not usable for AI and machine learning training. Webz.io publishes no user-agent string and no robots.txt token for it. If you do not want Webz.io to collect your pages, block all of its crawler names in robots.txt and on your server.

## What Webz.io documents

Webz.io sells web data through APIs. On July 17, 2024, it published the post [From Omgilibot to the Webzio Duo](https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/). The address `webz.io/bot.html` now leads to this post. The post describes two bots:

| Bot | What Webz.io says it does |
|---|---|
| Webzio | Crawls websites and structures the content |
| Webzio-extended | Takes the data that Webzio collected and tags it as usable or not usable for AI and machine learning training |

The post says that both bots obey robots.txt exclusions. It also says that they find content that is marked as off limits for AI use. For AI purposes, Webz.io says that it supplies only data that is tagged as explicitly permitted.

## What the post does not say

The post leaves out these facts:

- The full user-agent string of each bot
- The exact robots.txt token of each bot
- The signal that marks content as off limits for AI use
- Whether Webzio-extended sends its own requests or only tags data that Webzio fetched

A user agent is the client name in the User-Agent HTTP header. Public bot lists, such as [crawler-user-agents](https://github.com/monperrus/crawler-user-agents), record the Webzio crawler with this user agent:

```text
webzio (+https://webz.io/bot.html)
```

That list has no separate entry for Webzio-extended.

## Omgilibot, the older Webz.io crawler

Webz.io explained Omgilibot in a [2017 post](https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/). It built the bot for the Omgili search engine, which it later discontinued. The bot then collected pages for the Webz.io API, with a focus on news sites, blogs, reviews, ecommerce and discussion forums. The post gives this user agent:

```text
omgili/0.5 +https://omgili.com
```

Third-party bot lists use the name `omgilibot`. The post also says that Webz.io always complies when a site owner asks it to stop, through robots.txt or directly.

## Block Webz.io crawlers in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Because Webz.io does not document the tokens, list the names from the post and the older Omgilibot names in one group:

```text
User-agent: Webzio
User-agent: Webzio-extended
User-agent: omgili
User-agent: omgilibot
Disallow: /
```

Case does not matter. Under [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), crawlers match these names without regard to case.

If you want to keep your content out of AI training data only, add a group for `Webzio-extended` alone. This name matches the role that the post gives that bot:

```text
User-agent: Webzio-extended
Disallow: /
```

Webz.io does not confirm that this group has an effect.

## Block on the server

If you want to make sure that the requests stop, block them on your web server too. This nginx example returns HTTP 403 for any user agent that contains `webzio` or `omgili`:

```nginx
if ($http_user_agent ~* "(webzio|omgili)") {
    return 403;
}
```

Put the example in the `server` block of your site configuration, then reload nginx.

## Verify requests

Webz.io publishes no IP list and no reverse DNS method for its crawlers. Anyone can copy a user-agent string. A match in your logs does not prove that the request came from Webz.io.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/webzio-extended/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)