Grok by xAI: Chatbot, Web Search, Citations, Site Access
Grok is xAI's AI assistant. How it searches the web and X, how it cites sources, and why xAI documents no crawler or robots.txt token.
Grok is an AI assistant made by xAI. It can search the web and posts on X, and it can return the URLs of its sources. xAI publishes no documentation for a web crawler, a user-agent string or a robots.txt token. So a site owner has no documented way to allow or block Grok.
How Grok finds sources
xAI describes Grok’s search in its developer documentation. The tools overview lists two search tools that run on xAI servers:
| Tool | Description from xAI |
|---|---|
| Web Search | “Search the web and browse pages” |
| X Search | “Search X posts, users, and threads” |
The search tools guide says that Web Search lets Grok “search the web in real-time and browse web pages to find information.” xAI does not say which search index the tool uses.
A developer who calls the API can limit Web Search to at most five domains with allowed_domains. The developer can also exclude at most five domains with excluded_domains. These are settings for the developer, not for the owner of the site.
xAI publishes no technical page of this kind for the Grok app. The details on this page come from the API documentation.
The Grok chatbot
People chat with Grok in the app at grok.com. As of October 2026, grok.com describes Grok as “an AI assistant built by SpaceXSI” that gives “real-time answers from the web and X”. The xAI developer documentation also uses the name SpaceX Super Intelligence. In September 2025, Cloudflare wrote that “xAI’s bot, grok, does not self-identify at all, making it impossible for website operators to block it.”
How Grok cites sources
The citations documentation describes two outputs:
- A citations list with the URLs of all sources that Grok found during the search. xAI says that the final answer does not always use every URL in this list.
- Inline citations, which are Markdown links in the answer text at the place where the model uses a source. They have numbers, like
[[1]](url). xAI says that the model decides when to cite, so not every answer has them.
Posts on X appear as x.com URLs in these lists.
Crawlers and robots.txt
As of October 2026, xAI has no crawler page, no published IP list and no robots.txt token. Third-party bot lists show several different names for xAI bots, such as GrokBot and xAI-Grok, and these lists do not agree. The section below covers the name GrokBot.
A robots.txt rule for a name that xAI does not document has no documented effect. If you want to stop unknown AI traffic, block it on your server or firewall. Use the user agent or IP address from your own logs. A user-agent string is easy to copy. A match in your logs does not prove that a request came from xAI. Rate limits and bot protection services act on the behavior of requests, so they also work when a request uses a normal browser user agent.
The GrokBot name
Known Agents lists GrokBot as an undocumented xAI agent with the user agent Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/). The ai.robots.txt block list includes the tokens GrokBot and xAI-Grok. xAI does not document either name. Grok Bot, in two words, is a different thing: an xAI agent product that works in a browser on a cloud computer.
If you want to ask the reported names not to crawl your site, add two groups to robots.txt:
User-agent: GrokBot
Disallow: /
User-agent: xAI-Grok
Disallow: /