Kagi Search: Kagibot User Agent, Sources and robots.txt
Kagi builds its own index with Kagibot. Kagibot user agent, IP addresses, robots.txt rules with Googlebot fallback, and Kagi Small Web.
Kagi is a search engine that users pay for with a subscription. It builds its own web index with a crawler named Kagibot, and it also gets results from other search providers. Kagibot follows robots.txt. If your robots.txt has no rule for Kagibot, Kagibot follows your Googlebot rules.
Where Kagi results come from
Kagi’s search sources page lists these sources:
- Its own web index (internal name “Teclis”) and news index (internal name “TinyGem”).
- Anonymized API calls to “all major search result providers worldwide”, and to special search engines such as Marginalia.
- Sources for specific types of results, such as Wolfram Alpha, Apple, Wikipedia, Open Meteo, Yelp and TripAdvisor.
Kagi says that a typical query calls “a dozen or so different sources” at the same time. Kagi also says that its ranking puts pages with many ads and trackers lower.
Kagibot user agent and IP addresses
Kagi’s bot page gives this user-agent string:
Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot)The same page lists the IP addresses for Kagibot requests:
| IP address | Hostname |
|---|---|
| 216.18.205.234 | amd.kagibot.org |
| 35.212.27.76 | donna.kagibot.org |
| 104.254.65.50 | knet.kagibot.org |
| 209.151.156.194 | ogi.kagibot.org |
If a request claims to be Kagibot, make sure that its IP address is on this list. The list can change, so use the current version on the Kagi bot page.
Allow or block Kagibot in robots.txt
Kagi says that Kagibot follows the standard robots.txt rules that target it. This example from Kagi lets Kagibot crawl all pages except those under /private/:
User-Agent: Kagibot
Allow: /
Disallow: /private/If you want to block Kagibot from the whole site, use this rule:
User-Agent: Kagibot
Disallow: /If there is no Kagibot rule but there is a Googlebot rule, Kagibot follows the Googlebot rule. With the robots.txt below, Kagibot crawls all pages except /private/, even though the * group blocks everything:
User-Agent: *
Disallow: /
User-Agent: Googlebot
Allow: /
Disallow: /private/A Kagibot rule stops only the Kagi crawler. Kagi does not say whether it hides blocked pages that it gets from other providers.
How Kagi AI answers cite pages
Quick Answer writes a summary of the search results. Kagi says that it “provides references to the pages that are used.” A query that ends with a question mark starts Quick Answer by default.
Kagi Small Web and user rankings
Kagi Small Web shows recent posts from personal blogs in Kagi results and on kagi.com/smallweb. If you want to add a blog, open a pull request that adds its RSS or Atom feed to smallweb.txt in the smallweb repository. The rules include these:
- The blog is a personal blog in English with one author.
- The blog has a post from the last 12 months.
- The site has no ads, no undisclosed affiliate links and no popups.
- The content is not auto-generated or written by an LLM.
- If you submit your own site, also add at least two qualifying sites that are not yours.
Kagi users can block, lower, raise or pin a domain in their own results. Kagi publishes an anonymous total of these choices on its Personalization Leaderboard.