StableLM: Stability AI Open Models and Web Access
Table of Contents
StableLM is a family of open-weight language models from Stability AI. An open-weight model is a model whose files anyone can download and run. A StableLM model has no web access of its own. Stability AI documents no crawler or user agent for it, so a site owner has nothing to block.
The StableLM models
Stability AI published the first StableLM models in its StableLM GitHub repository. The repository and the model cards on Hugging Face give these releases:
| Model | Release | License |
|---|---|---|
| StableLM-Alpha base models, 3B and 7B | April 20, 2023 | CC BY-SA-4.0 |
| StableLM-3B-4E1T | September 29, 2023 | CC BY-SA-4.0 |
| Stable LM 2 1.6B | Technical report February 2024 | Stability AI Community License |
The README also says that the StableLM-Tuned-Alpha chat models use a non-commercial license (CC BY-NC-SA-4.0). For commercial use of Stable LM 2, the model card points to Stability AI’s license page.
Web access
A language model takes text in and writes text out. The Stable LM 2 1.6B model card describes a text generation model and lists no web tools. If an app that uses StableLM reads web pages, the app sends those requests. Look for the user agent of that app in your logs, not for a StableLM name.
Web data in training
Stability AI trained Stable LM 2 1.6B on public datasets, for example Falcon RefinedWeb, RedPajama-Data and CulturaX (model card). Falcon RefinedWeb is built from Common Crawl data (dataset card). Common Crawl collects pages with its crawler CCBot.
A block in robots.txt today does not remove your pages from models that are already trained. If you want to keep your pages out of future Common Crawl data, read the CCBot guide. For bots that do read the web for AI, see AI Crawler Bots: Training, Search and User Fetchers.