Consensus: AI Search for Research Papers and Sources
Consensus searches over 200 million research papers from Semantic Scholar, OpenAlex and its own crawl. How its search and ranking work.
Table of Contents
Consensus is an AI search engine for scientific research, run by Consensus NLP, Inc. It searches a corpus of research papers, not the general web, and writes a summary that cites the papers. A normal web page cannot appear in Consensus results. A research paper can appear if one of the Consensus sources has it.
Sources
The help article Welcome to Consensus says that the corpus holds over 200 million scientific documents from all fields of science. Most are peer-reviewed journal articles. Some are conference papers and preprints. Consensus builds the corpus from three sources:
- Semantic Scholar
- OpenAlex, an open catalog of scholarly works
- Its own crawl of the scholarly web, to fill gaps in coverage
Consensus says that the combined sources cover nearly all of the highest-impact journals and all of PubMed. Consensus does not name the crawler for its own crawl. It publishes no user agent, no robots.txt token and no IP list for it.
How a search works
The same article describes the search in three steps:
- Consensus matches the query against the title and abstract of every paper. It combines semantic search, which uses AI embeddings to match meaning, with BM25 keyword search, a standard method that matches exact terms.
- It takes the top 1,500 papers and ranks them again. This step adds quality signals: how recent the paper is, its citation count, and the reputation and impact of the journal.
- A larger AI model scores relevance again for the top 20 papers. The quality signals still count in this final ranking.
The AI then reads the top results and writes a summary. Every response cites its papers, so a reader can go to each source.
Consensus says that it is a search engine, not a chatbot. Its best practices article says that it is not built for general knowledge questions. It works best for topics that peer-reviewed literature covers.
What it means for authors and site owners
A paper enters Consensus through Semantic Scholar, OpenAlex or the Consensus crawl. The help article names no form to submit a paper. If a paper is missing, make sure that Semantic Scholar or OpenAlex lists it. The first search step reads only titles and abstracts.
Consensus does not index general web pages. So there is nothing to block for a normal website, and Consensus documents no opt-out.