Dolma is an open, three-trillion-token English text dataset from the Allen Institute for AI, used to pretrain its OLMo language models.
In plain words
Dolma is a documented mix of web pages, research papers, program code, old books now in the public domain, social posts and encyclopedia articles, released so researchers can study what a model learns from. The team also open-sourced the toolkit used to build it, so others can reproduce or adapt the process.
A quick example
Imagine a researcher who wants to know whether removing duplicate web pages helps a model. They can download Dolma, run the open Dolma toolkit with different settings, and train small models to compare, all without collecting data from scratch.
Where you'll meet it
- Pretraining AI2's OLMo models.
- Research on how pretraining data is curated.
Related
Sources
- PaperDolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, Soldaini et al. (arXiv) · read 29 Sept 2026
- Docsallenai/dolma, Allen Institute for AI (GitHub) · read 29 Sept 2026
- Docsallenai/dolma dataset card, Allen Institute for AI (Hugging Face) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.