Data curation is choosing, cleaning and filtering training data so a model learns from better examples; a model's quality depends heavily on its data.
In plain words
Raw data from the web is full of repeats, spam and low-quality text. Curation decides what to keep: removing duplicates, filtering out junk, and sometimes favouring certain kinds of content. Researchers test these choices by training small models on different versions of a dataset and comparing the results.
A quick example
The FineWeb team started from 96 Common Crawl web snapshots, tried different deduplication and filtering rules, and kept the choices that gave better models. A further filter pulled out educational text to make a smaller set called FineWeb-Edu.
Where you'll meet it
- Building pretraining datasets for language models.
- Image-text datasets for models such as CLIP.
Related
Sources
- PaperThe FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, Penedo et al. (arXiv) · read 29 Sept 2026
- PaperDataComp: In search of the next generation of multimodal datasets, Gadre et al. (arXiv) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.