C4

C4, the Colossal Clean Crawled Corpus, is a large, cleaned-up set of English web text made from Common Crawl, first built to train Google's T5 model.

In plain words

Raw web pages contain a lot of messy text. The T5 team cleaned Common Crawl data into a large collection of English text and called it C4. They released it along with T5 in 2019. A processed copy is hosted on Hugging Face in several variants, including an uncleaned one and a multilingual one, mC4, covering over 100 languages.

A quick example

Imagine a team testing a new small language model. They might pre-train it on the English C4 variant, so they can compare their results fairly with other papers that used the same data.

Where you'll meet it

• Pre-training language models such as T5.
• A shared baseline dataset for research.

Related

Sources

1. PaperExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Raffel et al. (arXiv) · read 29 Sept 2026
2. Docsallenai/c4, Hugging Face · read 29 Sept 2026
3. Docsc4, TensorFlow Datasets · read 29 Sept 2026

Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.