Llama Models by Meta: Web Access and Training Data

Llama models and licenses
Web search is a tool call
Training data and Meta crawlers

Llama is a family of open-weight language models from Meta. Open-weight means that anyone can download the model files and run them. A Llama model has no web access of its own and sends no requests to your site. The app that runs the model does any web fetching.

Llama models and licenses

Meta publishes the model cards, prompt formats and licenses in the llama-models repository on GitHub. The Llama 4 model card gives April 5, 2025 as the release date of Llama 4 Scout and Llama 4 Maverick. Each model version has its own community license, for example the Llama 4 Community License Agreement.

Meta also uses Llama in its own assistant. Meta announced that assistant in April 2024 as “Meta AI, Built With Llama 3”. For how Meta AI reads the web, see the Meta AI guide.

Web search is a tool call

Llama 3.1 has built-in tools. The Llama 3.1 prompt format lists three of them: Brave Search, Wolfram Alpha and a code interpreter. When Brave Search is turned on, the model can answer with a tool call like this:

brave_search.call(query="latest price of 1oz gold")

The model then waits for a message with the output of the tool call. The model does not connect to the web itself. The software of the developer runs the search, fetches any pages and sends the result back to the model. Requests to your site come from that software, with its own user agent and IP address.

Training data and Meta crawlers

The Llama 3.1 model card says that Llama 3.1 was pretrained on data from publicly available sources. Its pretraining data has a cutoff of December 2023. The Llama 4 model card says that Llama 4 used publicly available data, licensed data and information from Meta products and services. Its cutoff is August 2024.

The model cards do not name the crawler that collected the public data. The Meta web crawlers page lists Meta-ExternalAgent for the training of foundation AI models. If you do not want Meta to collect your pages for this use, add this group to your robots.txt file:

User-agent: meta-externalagent
Disallow: /

A new rule does not change Llama models that Meta already released.