BuiltWith Bot: User Agent, robots.txt and Removal
BuiltWith visits sites to detect their technology stack. Its user agent values, the robots.txt rule it follows and how to remove your site profile.
The BuiltWith bot visits websites to find out which technologies they use. BuiltWith shows the results as public technology profiles. It also offers the data in paid tools for lead generation, sales intelligence and market analysis. If you do not want to be profiled, disallow BuiltWith in robots.txt and remove your existing profile.
What BuiltWith collects
BuiltWith explains its crawler in the BuiltWith Indexing Usage Policy. The bot looks at the publicly visible code of a website to build its technology profile. BuiltWith says that anyone can find the same information without its tool.
The policy also lists what BuiltWith does not do:
- It does not store or cache any part of your website.
- It does not copy or resell your website content.
- It does not try to log in or to open restricted areas of your website.
User agent
BuiltWith states that all its requests contain a user agent that includes one of these values:
| Status | Value in the user agent |
|---|---|
| Current | BuiltWith/1.4; rb.gy/xprgqj |
| Current | BuiltWith/1.5; h1.nu/1ySUD |
| Historical | BW/1.1; rb.gy/oupwis |
| Historical | BW/1.3; rb.gy/qyzae5 |
The robots.txt token is BuiltWith. The policy does not publish IP ranges or a DNS verification method.
BuiltWith also publishes a key directory at https://builtwith.com/.well-known/http-message-signatures-directory. This is the format of Web Bot Auth, a method in which a bot signs each request with a private key. A service that supports Web Bot Auth, such as Cloudflare, can use the public key to confirm signed requests from BuiltWith.
Block BuiltWith in robots.txt
If you want BuiltWith to stop indexing your site, add this group to the robots.txt file at the root of your domain:
User-agent: BuiltWith
Disallow: /BuiltWith states that it reads robots.txt when it indexes a site and stops if it finds this rule. A User-agent: * group with Disallow: / also stops it, but that rule blocks all compliant crawlers.
If you also block the bot on your server or firewall, match both BuiltWith/ and BW/ in the user agent.
Remove your site from BuiltWith profiles
A robots.txt rule stops new visits. To remove the public profile, use the BuiltWith removals page. You must own the domain. Follow these steps:
- Enter the root domain of your site on the removals page.
- Copy the removal code that the page shows.
- Create a text file at
/builtwith.txton your site that contains the code.
The file confirms that you own the domain. If you cannot create a file on your web server, BuiltWith asks you to contact it.
For a similar technology detection tool, see the Wappalyzer guide.