Elasticsearch: Self-Hosted Search and the Elastic Crawler

Elasticsearch is search software from Elastic. A company runs it on its own servers or on Elastic Cloud to search its own data. Examples are the pages of its website or its product catalog. It is not a public AI search engine, so you cannot submit a site to it. If you see Elastic-Crawler in your logs, someone is indexing your pages into their own Elasticsearch. If you do not want that, block it in robots.txt.

What Elasticsearch is used for

Elastic’s search solutions page lists uses such as website and documentation search, ecommerce product catalogs, recommendations and question answering. It also lists RAG (retrieval augmented generation), where an AI model answers from retrieved documents. The same page says that Elasticsearch can act as a vector database. A vector is a list of numbers that represents the meaning of a text.

On August 29, 2024, Elastic added the AGPL license as an option for Elasticsearch, next to the ELv2 and SSPL licenses.

Semantic search

Elastic’s semantic search page says that semantic search uses natural language processing models and vector search to find results by meaning, not just keywords. It gives three ways to set it up:

1. The semantic_text field type, which Elastic recommends as the simplest way.
2. The inference API, for more control over the model endpoint.
3. Models deployed directly in Elasticsearch with the Eland tool, the most complex way.

Elastic Open Crawler

Elastic Open Crawler is a crawler that collects web content and indexes it into Elasticsearch. The operator runs it from the command line, usually in Docker, with a configuration file. It works with Elasticsearch 8.x and 9.x. It is self-hosted, so its requests come from the operator’s servers, not from Elastic.

The example configuration gives the default user agent as Elastic-Crawler followed by the version in parentheses, for example:

Elastic-Crawler (0.3.0)

The operator can set a custom user agent with the user_agent setting. Elastic publishes no IP list for the crawler.

Block or allow Elastic Open Crawler

The crawler directives page says that the crawler always follows robots.txt. The operator cannot turn this off. It also follows robots meta tags, nofollow links, canonical URL tags and sitemaps. It does not support Crawl-delay.

The crawler matches robots.txt groups against its user agent, which is Elastic-Crawler by default. If you want to block it, add this group:

User-agent: Elastic-Crawler
Disallow: /

If an operator sets a custom user agent, this group does not match. The crawler then follows the group for its custom name or the User-agent: * group.

A block stops only this crawler. It does not change how public search engines see your site.