# WappalyzerBot: Wappalyzer Crawler User Agent and robots.txt

> WappalyzerBot crawls public pages to detect the technologies a site uses. Its user agent, signed requests, robots.txt rules and how to block it.

WappalyzerBot is the crawler of Wappalyzer, a service that identifies the technologies that websites use. It is not an AI crawler. Wappalyzer says that its signed crawls follow robots.txt rules for `WappalyzerBot`. If you do not want the bot on your site, block it in robots.txt.

## What WappalyzerBot does

Wappalyzer describes the bot on its [crawler page](https://www.wappalyzer.com/crawler/). Wappalyzer Pty Ltd operates it. The bot fetches public pages and reads signals such as HTML, scripts, response headers, cookies and page metadata. From these signals, Wappalyzer identifies the technologies that a site uses.

The crawls keep the Wappalyzer technology dataset current. They also run when someone requests a technology lookup in a Wappalyzer product. Both kinds of crawl use the same crawler and the same identification. Each crawl has limits on the number of URLs, the link depth and the total time. Requests for linked pages are delayed.

## User-agent string

A user agent is the client name in the User-Agent HTTP header. Wappalyzer documents this user agent:

```text
WappalyzerBot/1.0 (+https://www.wappalyzer.com/)
```

## Block WappalyzerBot in robots.txt

robots.txt is a text file at the root of your site with rules for crawlers. Wappalyzer says that signed crawls read robots.txt before they fetch a page. They follow the `Allow`, `Disallow` and `Crawl-delay` rules for `WappalyzerBot`. A signed crawl stops if it cannot read the rules, or if the crawl delay does not fit in its time limit.

Wappalyzer gives this group to block the bot:

```text
User-agent: WappalyzerBot
Disallow: /
```

If you want to slow the bot down instead, put a `Crawl-delay` line in the group, for example `Crawl-delay: 10`. If you see unexpected crawler traffic, Wappalyzer asks you to email `hello@wappalyzer.com`. Include the hostname and the time of an example request.

## Verify requests

Signed WappalyzerBot requests use HTTP Message Signatures ([RFC 9421](https://datatracker.ietf.org/doc/rfc9421/)). An HTTP message signature is a cryptographic signature in the request headers. Wappalyzer publishes the public keys in its [signature directory](https://api.wappalyzer.com/.well-known/http-message-signatures-directory). If your server or CDN supports these signatures, use the keys to make sure that a request came from Wappalyzer. The crawler page does not say whether every request is signed. Wappalyzer publishes no IP list.

## What a block does not cover

The Wappalyzer [browser extension](https://www.wappalyzer.com/faq/extension/) works in a different way. It inspects HTML, scripts, cookies, network requests and certificates on the pages that a person opens in their own browser. By default, it can send website data, such as the hostname and the detected technologies, to Wappalyzer. Users can turn this off in the extension settings. These are normal visits by a person, so robots.txt rules for WappalyzerBot do not apply to them.

Wappalyzer says that it is often impractical to hide the technologies of a site. For security, it recommends that you keep your software up to date. [BuiltWith](/ai-crawler-bot/builtwith/) is another service that profiles website technologies with its own bot.

---

[View the canonical page](https://aichatwatch.com/ai-crawler-bot/wappalyzer/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)