Semantic Scholar: Ai2 Paper Search and Its Sources

Semantic Scholar is a free search engine for research papers, run by Ai2 (the Allen Institute for AI). It indexes academic papers from publisher partners, data providers and web crawls, not general web pages. If you publish research, claim your author page, and if you are a publisher, ask about a direct partnership.

Who runs Semantic Scholar

The About page says that Semantic Scholar launched in 2015 as a project at Ai2. Ai2 is a non-profit research institute that Paul Allen founded in 2014. The same page says that Semantic Scholar indexes over 200 million academic papers.

Where the papers come from

The Semantic Scholar FAQ says that it gets content through web indexing and through partnerships with journals, indexes and content providers. It names PubMed, arXiv and Springer Nature as examples. The publisher partners page lists more than 50 direct partnerships with publishers, data providers and aggregators.

The FAQ also states the limits of the corpus:

• It focuses on published academic articles and preprints.
• Book coverage is very limited.
• Patents are not included.

Semantic Scholar documents no crawler name, user agent or robots.txt token for its web indexing.

AI features

Semantic Scholar shows a TLDR on some papers. A TLDR is a one-sentence summary that an AI model writes. The FAQ says that TLDRs cover only computer science and biomedical papers. Search results link to paper pages on semanticscholar.org, with access links to the publisher or to a PDF when one is available.

The paper data is open to developers through the Semantic Scholar API and the S2AG and S2ORC datasets. Other research tools build on it. For example, Undermind searches the Semantic Scholar database.

For authors and publishers

• If you are an author, claim your author page. Then you can add missing papers and remove papers that are not yours. Changes take about 24 hours.
• If you want a paper or PDF removed from the site, use the Contact Us form at the bottom of the FAQ page. Select the topic Remove A Paper, and give the paper URL and the reason.
• If you are a publisher, read the publisher partners page about adding your content to the Semantic Scholar knowledge graph.

AI2Bot is a different crawler

Ai2 also runs a crawler called AI2Bot. Its crawler page says that the content it collects trains open language models. The page does not mention Semantic Scholar. Ai2 does not say whether a block of AI2Bot has an effect on Semantic Scholar. For the user agent and blocking steps, see the AI2Bot guide.