A/B testing an LLM app means showing two versions, such as two prompts or models, to different real users and comparing which one performs better.
In plain words
Tests on a fixed dataset only show part of the picture. In an A/B test, each request is randomly given version A or version B, and the results are tracked: quality scores, cost, token use and latency. It works best after offline testing, on a small group of users first, in apps where occasional weaker answers are not a big deal.
A quick example
Imagine a writing assistant testing a shorter system prompt. Half of the requests use the old prompt, labelled prod-a, and half use the new one, prod-b. After a week the team compares user ratings, cost per request and latency for each, then keeps the winner.
Where you'll meet it
- Comparing prompt versions in production.
- Checking whether a cheaper model keeps quality.
Related
Sources
- DocsA/B Testing of LLM Prompts, Langfuse documentation · read 29 Sept 2026
- DocsPrompt Registry, MLflow documentation · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.