BIG-bench is a large, community-built benchmark of more than 200 language tasks, designed to probe what language models can and cannot yet do.
In plain words
Instead of one kind of test, BIG-bench gathers hundreds of tasks written by hundreds of researchers, from logic puzzles to language games. Many were chosen because models were expected to fail them. Human experts also did the tasks to give a baseline. A smaller subset, BIG-bench Lite, is cheaper to run.
A quick example
Imagine a task asking models to spot the logical error in a short argument. Researchers can run models of many sizes on it and see whether bigger models improve steadily, suddenly, or not at all.
Where you'll meet it
- Studying how abilities change as models grow.
- Comparing language models in research.
Related
Sources
- PaperBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, Srivastava et al. (arXiv) · read 29 Sept 2026
- DocsBIG-bench, Google (GitHub) · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.