LLM evaluation

1 · What it is

A language model can give different replies to the same input, and that makes the usual ways of testing software fall short. So teams build evaluations, or “evals”: structured tests of how accurate and reliable an AI feature is.

This page is about LLM evaluation: testing features built on large language models.

Picture a bike shop chatbot. You want to try a new prompt, the written instructions the model gets before each chat. Is the new version better than the old one on real questions?

Start with real cases. Good test data can come from real chats with customers, from people who know the subject, and from old records. Mix everyday questions, edge cases and adversarial cases, which are questions written on purpose to trick the bot. An edge case is an unusual but possible input.

Say what passing means. A pass rule should be specific and measurable. Anthropic’s guide gives a loose goal as a warning: a model that should “classify sentiments well”, which roughly means be good at spotting happy and angry reviews. A tighter rule for the bike shop names the exact thing to look for: the answer states the return window from the policy page and does not invent one.

Choose a grader. A code check can compare the output with a known correct answer or look for a key phrase. Code checks are the fastest and most reliable kind of grading, but they miss the finer points.

When a known correct answer exists, fixed scoring formulas with names like ROUGE or BLEU can compare the output with it.

For fuzzier questions, a second model can grade the first. This is called an LLM judge. It is fast and handles harder calls, but test it before you rely on it. The judge needs a detailed, clear rubric: a marking sheet that spells out what earns a pass.

Google Cloud’s guide compares rubrics to unit tests. A unit test is a tiny automatic check a programmer writes for one small piece of code. A rubric does the same job for an answer: a short list of things that must be true. Some tools can even draft a custom checklist of yes-or-no tests for each prompt, which Google calls adaptive rubrics.

Can you trust a judge model? A 2023 research paper built a question set called MT-Bench and compared judge models with people. Strong LLM judges agreed with human preferences over 80% of the time. That is about as often as two humans agree with each other. Judge models showed biases, such as favouring an answer because of its position or its length.

OpenAI’s guidance says to use people’s own marks to check and adjust the automatic scores. Check a judge model agrees with people first; make it cheaper or faster later.

A worked example. Say the bike shop has twenty test questions, including “How do I return a helmet?” In the bike shop test, the code check is: does the answer contain the right return window? The judge follows a marking sheet for whether the tone is polite and the answer is clear. If it fails a case the old prompt passed, that is a regression: a step backwards. Regression tests can require a new version to beat the old one on the measures you care about.

Compare and trace. A trace is a record of all the in-between steps of one request, not just the final answer. When a case fails, you open its trace and see where it went wrong.

Offline and online. Offline evaluation is testing before launch. Online evaluation tracks quality on live traffic after launch. The two feed each other: issues found in live traffic can be added to the offline test set. Run evals on every change to catch a prompt that quietly makes things worse.