An evaluation harness is a tool that runs a model through many test tasks in the same way every time and scores the results, so models can be compared fairly.
In plain words
Each evaluation bundles a dataset of questions, the way the model is prompted or run, and a scorer that marks the answers. The harness handles the repetitive parts: loading tasks, calling the model, collecting outputs and reporting scores. EleutherAI's LM Evaluation Harness runs many standard academic benchmarks, and the UK AI Security Institute's Inspect adds building blocks for testing agents that use tools.
A quick example
A team fine-tunes an open model and wants to know if it got worse at general knowledge. They run the same benchmark tasks through the harness on the old and new versions and compare the scores side by side.
Where you'll meet it
- Comparing models on public benchmarks.
- Regression tests before releasing a new model version.
Related
Sources
- Docslm-evaluation-harness, EleutherAI (GitHub) · read 29 Sept 2026
- DocsInspect, UK AI Security Institute · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.