Common Crawl is a non-profit that regularly crawls the web and publishes the pages as a free, open archive, and cleaned-up versions of it, such as C4, are used as text datasets for AI.
In plain words
Its crawler, CCBot, visits websites, checks each site's robots.txt rules first, and saves the pages it is allowed to fetch. Crawls have been collected since 2008, and the archive now holds petabytes of data. Anyone can download it. Raw web pages are messy, so datasets built from it, such as C4, filter and clean the pages first.
A quick example
A research team wants a large English text set. It downloads a recent Common Crawl snapshot, keeps only pages in English, removes duplicates and boilerplate, and uses the rest as training text.
Where you'll meet it
- Raw material for cleaned text datasets such as C4.
- Web research and analysis.
Related
Sources
- DocsCommon Crawl, Common Crawl · read 29 Sept 2026
- DocsOverview, Common Crawl · read 29 Sept 2026
- DocsFAQ, Common Crawl · read 29 Sept 2026
- DocsC4, TensorFlow Datasets · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.