# Kagi Search: Kagibot User Agent, Sources and robots.txt

> Kagi builds its own index with Kagibot. Kagibot user agent, IP addresses, robots.txt rules with Googlebot fallback, and Kagi Small Web.

Kagi is a search engine that users pay for with a subscription. It builds its own web index with a crawler named Kagibot, and it also gets results from other search providers. Kagibot follows robots.txt. If your robots.txt has no rule for Kagibot, Kagibot follows your Googlebot rules.

## Where Kagi results come from

Kagi's [search sources page](https://help.kagi.com/kagi/search-details/search-sources.html) lists these sources:

- Its own web index (internal name "Teclis") and news index (internal name "TinyGem").
- Anonymized API calls to "all major search result providers worldwide", and to special search engines such as Marginalia.
- Sources for specific types of results, such as Wolfram Alpha, Apple, Wikipedia, Open Meteo, Yelp and TripAdvisor.

Kagi says that a typical query calls "a dozen or so different sources" at the same time. Kagi also says that its ranking puts pages with many ads and trackers lower.

## Kagibot user agent and IP addresses

Kagi's [bot page](https://kagi.com/bot) gives this user-agent string:

```
Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot)
```

The same page lists the IP addresses for Kagibot requests:

| IP address | Hostname |
|---|---|
| 216.18.205.234 | amd.kagibot.org |
| 35.212.27.76 | donna.kagibot.org |
| 104.254.65.50 | knet.kagibot.org |
| 209.151.156.194 | ogi.kagibot.org |

If a request claims to be Kagibot, make sure that its IP address is on this list. The list can change, so use the current version on the Kagi bot page.

## Allow or block Kagibot in robots.txt

Kagi says that Kagibot follows the standard robots.txt rules that target it. This example from Kagi lets Kagibot crawl all pages except those under `/private/`:

```
User-Agent: Kagibot
Allow: /
Disallow: /private/
```

If you want to block Kagibot from the whole site, use this rule:

```
User-Agent: Kagibot
Disallow: /
```

If there is no Kagibot rule but there is a Googlebot rule, Kagibot follows the Googlebot rule. With the robots.txt below, Kagibot crawls all pages except `/private/`, even though the `*` group blocks everything:

```
User-Agent: *
Disallow: /

User-Agent: Googlebot
Allow: /
Disallow: /private/
```

A Kagibot rule stops only the Kagi crawler. Kagi does not say whether it hides blocked pages that it gets from other providers.

## How Kagi AI answers cite pages

[Quick Answer](https://help.kagi.com/kagi/ai/quick-answer.html) writes a summary of the search results. Kagi says that it "provides references to the pages that are used." A query that ends with a question mark starts Quick Answer by default.

## Kagi Small Web and user rankings

Kagi Small Web shows recent posts from personal blogs in Kagi results and on kagi.com/smallweb. If you want to add a blog, open a pull request that adds its RSS or Atom feed to `smallweb.txt` in the [smallweb repository](https://github.com/kagisearch/smallweb). The rules include these:

- The blog is a personal blog in English with one author.
- The blog has a post from the last 12 months.
- The site has no ads, no undisclosed affiliate links and no popups.
- The content is not auto-generated or written by an LLM.
- If you submit your own site, also add at least two qualifying sites that are not yours.

Kagi users can [block, lower, raise or pin a domain](https://help.kagi.com/kagi/features/website-info-personalized-results.html) in their own results. Kagi publishes an anonymous total of these choices on its Personalization Leaderboard.

---

[View the canonical page](https://aichatwatch.com/ai-search-engine/kagi-search/) · [Browse llms.txt](https://aichatwatch.com/llms.txt)