DeepEval is an open-source Python framework for unit testing LLM apps, a bit like Pytest, with ready-made metrics such as answer relevancy and hallucination.
In plain words
DeepEval lets you write test cases with an input and the app's output, pick metrics, and assert that scores pass a threshold. Many metrics use an LLM as a judge. Tests run locally with a command that works the way pytest does, and it fits agents, RAG pipelines and chatbots built with many frameworks.
A quick example
Imagine a RAG chatbot about a product manual. A developer writes a test case with a question and the bot's answer, adds a relevancy metric and a hallucination metric, and runs the test suite. If a code change makes answers drift from the manual, the test fails.
Where you'll meet it
- Unit tests for LLM features in CI.
- Evaluating RAG pipelines and agents.
Related
Sources
- Docsconfident-ai/deepeval, Confident AI (GitHub) · read 19 Oct 2026
- DocsIntroduction to DeepEval, DeepEval documentation · read 29 Sept 2026
- DocsDeepEval quickstart, DeepEval documentation · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 2 Nov 2026. Spotted an error? Report it.