Amazon Q: Web Crawler, robots.txt and Site Access

Amazon Q is the name of two AWS assistants: Amazon Q Developer for software work and Amazon Q Business for company data. AWS does not document a public web search feature for either one. A site owner meets Amazon Q mostly through the Amazon Q Business Web Crawler, which obeys robots.txt rules for the amazon-QBusiness token.

Amazon Q Developer

AWS describes Amazon Q Developer as a conversational assistant that helps you build and operate AWS applications. In an IDE (a code editor), it gives inline code suggestions, chat about code, code generation and security scans. It runs on Amazon Bedrock.

AWS will end support for the Amazon Q Developer IDE plugins on April 30, 2027. AWS points IDE users to Kiro, which offers agentic coding, chat and MCP support.

AWS does not document a crawler or a user agent that Amazon Q Developer uses to visit public websites. Its suggestions can be similar to public open-source code. In that case, the code reference log shows information about the source.

Amazon CodeWhisperer

Amazon CodeWhisperer was an AWS assistant that suggested code in a code editor. It became part of Amazon Q Developer on April 30, 2024, as the AWS Toolkit for JetBrains documentation states. The change includes inline code suggestions and security scans. The CodeWhisperer user guide says that customers can migrate in place and keep their subscriptions, customizations and tags. The old product page at aws.amazon.com/codewhisperer/ now redirects to the Amazon Q Developer page. AWS documents no crawler, user agent or robots.txt token for CodeWhisperer.

Amazon Q Business and its web crawler

Amazon Q Business answers questions from a company’s data sources and gives citations to those sources. AWS says that it is no longer open to new customers. Existing customers still get bug fixes and security updates, and AWS recommends Amazon Quick for new projects.

One of its data sources is the Web Crawler connector. A customer gives it website addresses, and it crawls and indexes those sites. AWS tells customers to crawl only their own pages or pages that they have permission to index.

Block the Amazon Q Business crawler

robots.txt is a text file at the site root with rules for crawlers. AWS says that the Amazon Q Business Web Crawler obeys the Allow and Disallow rules in robots.txt. If you want to stop it on your whole site, add this group:

User-agent: amazon-QBusiness
Disallow: /

If you want to keep it out of one folder only, disallow that path:

User-agent: amazon-QBusiness
Disallow: /credential-pages/

Amazon Quick web crawler

The Amazon Quick Web Crawler identifies itself with the user-agent string amazon-Quick-on-behalf-of-<UUID>. AWS says that it obeys robots.txt user-agent, allow and disallow rules. If it cannot get your robots.txt file, it crawls as if the file does not exist.

Other Amazon bots

Amazonbot, Amzn-SearchBot and Amzn-User are crawlers for Amazon products and services such as Alexa. The Amazonbot page does not connect them to Amazon Q. The Amazonbot guide explains how to control them.