Tavily: Search API for AI Agents and Site Owners
Tavily is a search, extract and crawl API for AI agents, now joining Nebius. It documents no crawler user agent or robots.txt token.
Tavily is a web search API for AI agents and apps that use large language models. Developers call it to search the web, extract the text of pages, and crawl or map whole sites. Tavily documents no crawler name, user agent or robots.txt token, so a site owner has no documented way to target it in robots.txt.
What Tavily offers
The Tavily documentation calls Tavily the web layer for AI agents. The API has these endpoints:
| Endpoint | What it does, as documented |
|---|---|
| Search | Runs a web search and returns ranked results with text snippets |
| Extract | Gets clean content from one or more URLs |
| Crawl | Follows links across a site and extracts the pages |
| Map | Finds the structure and URLs of a site |
| Research | Returns a report with citations from many sources |
Tavily also has an MCP server, a command-line tool and integrations with agent frameworks such as LangChain and LlamaIndex. MCP is a standard way to connect tools to AI assistants.
How results reach an AI answer
Each search result has a URL, a title, a short content snippet and a relevance score. The developer can also ask for the full cleaned page text and the favicon. The app that calls Tavily decides how to show or cite these URLs.
Developers can limit results to a list of domains with include_domains, or remove domains with exclude_domains. The FAQ says that Tavily gathers data from trusted, authoritative sources. It does not name a crawler or an index.
Ownership
In February 2026, Tavily announced that it is joining Nebius. The post says that the API stays the same.
What a site owner can do
- Tavily documents no user agent, IP address list or opt-out for its search, extract or crawl requests.
- The Crawl best practices in the documentation tell developers to respect the robots.txt file of a site and to add delays between requests. This is advice to API users. The documentation does not say that Tavily enforces robots.txt.
- If you see unwanted automated traffic, find the user agent and IP addresses in your server logs. Then block them at the server or firewall. For the steps, see how to find and block crawlers on your site.
For a similar search API, see Exa.