Falcon LLM by TII: Web Access and Common Crawl Data
Falcon is a family of open language models from the Technology Innovation Institute (TII) in Abu Dhabi. Falcon is a model, not a chat service with web access. It does not crawl, search or fetch web pages by itself, so there is no Falcon user agent to block. Your pages can be in its training data through Common Crawl.
What Falcon is
TII is the applied research part of the Advanced Technology Research Council (ATRC) of Abu Dhabi. The Falcon website lists models from Falcon 40B and Falcon 180B to Falcon 2, Falcon 3, Falcon-H1, Falcon Arabic and Falcon-E. Developers download the models, for example from Hugging Face, and run them in their own apps. The Falcon website does not mention a web search or browsing feature.
If an app that uses Falcon reads web pages, the app sends those requests with its own user agent. Identify such a bot by that user agent and its IP address, not by the Falcon name.
Common Crawl data in Falcon training
TII published RefinedWeb, a web dataset for training Falcon models. The RefinedWeb dataset card gives these facts:
- RefinedWeb is built from Common Crawl dumps.
- TII filtered and deduplicated the pages. It removed adult content with a URL blocklist and a score system.
- The public extract has about 968 million web pages.
- Falcon-7B and Falcon-40B were trained on RefinedWeb together with other selected data.
Common Crawl collects its pages with its crawler, CCBot. The dataset card says that this crawler follows opt-out requests in robots.txt.
Keep your pages out of future datasets
If you do not want Common Crawl to collect your pages, add this group to your robots.txt file:
User-agent: CCBot
Disallow: /This rule stops future Common Crawl collection. Common Crawl does not say that it removes your pages from crawl data that it already published.
RefinedWeb has its own removal process. If you want to remove a document from RefinedWeb, send a message to [email protected]. The dataset card gives this address for removal requests.