Algolia: Hosted Site Search and the Algolia Crawler

Algolia is a hosted search service that a company adds to its own website or app. It is not a public web search engine: people do not use it to search the whole web. Algolia matters to a website owner in two cases. You use it as the search box of your own site, or the Algolia Crawler visits your site.

What Algolia is

Algolia describes itself as a search-and-discovery platform with hosted APIs for full-text search, AI semantic search (NeuralSearch), recommendations and merchandising. Algolia says that NeuralSearch combines keyword search with vector search. Vector search finds results by meaning, not only by exact words.

A company puts its content into an Algolia index. An index is the searchable copy of the content. The company sends the data with the Algolia API clients or with the Algolia Crawler. Visitors then search that index through the search box on the site or app of the company.

The Algolia Crawler

The Algolia Crawler gets content from a site and sends it to the Algolia indices of the site owner. It starts from a list of URLs, follows links to find more pages, and can run on a schedule. For documentation sites, Algolia offers DocSearch, which uses the Crawler and includes a search interface.

An Algolia customer can only crawl domains that it owns. Algolia asks the customer to verify each domain with one of these methods:

• An email address on the same domain
• A meta tag on the home page
• An HTML file on the server
• A DNS TXT record with the host algolia-site-verification
• A comment in robots.txt, for example # Algolia-Crawler-Verif: XXXX

User agent and robots.txt

Algolia support says that the Crawler uses the user agent Algolia Crawler/xx.xx.xx, where xx.xx.xx is a version number that changes. In robots.txt, use the token without the version:

User-agent: Algolia Crawler
Allow: /

The Crawler does not follow a URL that robots.txt blocks. Algolia shows this as a “Forbidden by robots.txt” error. The owner of a crawler can turn this off with the ignoreRobotsTxtRules setting, which is false by default.

Firewall rules for the Crawler

The Algolia troubleshooting page gives the IP address of the Crawler as 34.66.202.43. If your firewall blocks the Crawler, add this address to its allowlist. Algolia support recommends an IP allowlist instead of user-agent rules in nginx or other custom checks.

If you do not want the Crawler on your site, add a Disallow: / rule for Algolia Crawler. The owner of a crawler can turn off robots.txt rules, so a firewall rule for this IP address gives a stronger block.