9.4Part 3: LLM Evals -- Testing the AI Inside Your Product
When the product embeds an LLM, assert response == expected is dead on arrival: the same input can produce different valid outputs. The industry answer is evals -- test suites where each case is scored (by metrics, heuristics, or a judge model) and the suite passes on aggregate thresholds.
The Mental Model: Evals Are Parametrized Tests with Scored Assertions
| Classical test suite | LLM eval suite |
|---|---|
| Test case: input + expected output | Eval case: input + reference answer (or scoring rubric) |
| Assertion: exact match / predicate | Scorer: metric or LLM-as-judge, produces 0.0-1.0 |
| Pass/fail per test | Score per case, threshold per suite ("faithfulness >= 0.85") |
| Regression = a test turns red | Regression = a metric drops vs. the previous model/prompt version |
| Run on every PR | Run on every prompt change, model upgrade, retrieval-index rebuild |
The Frameworks to Know
| Framework | What It Is | When You Reach for It |
|---|---|---|
| Ragas | Open-source eval library focused on RAG pipelines: faithfulness, answer relevancy, context precision/recall | Your feature retrieves documents and generates answers from them |
| TruLens | Instrumentation + eval: traces LLM app internals, applies "feedback functions" (groundedness, relevance) to each step | You need to see where in a chain quality is lost, not just the final score |
| OpenAI Evals | Framework + registry for building eval suites against models/prompts, including model-graded (LLM-as-judge) evals | Building custom eval suites and CI-style regression runs on prompts |
You do not need deep expertise in all three. You need to be able to say what each is for, and to design an eval suite in one of them: pick the metric, build the golden dataset (20-100 curated cases beats 10,000 scraped ones), set the threshold, wire it into CI so a prompt change that drops faithfulness below threshold blocks the merge -- exactly like a coverage gate.
LLM-as-Judge, Reviewed Like AI Tests
Model-graded evals use a strong model (as of July 2026: Claude Opus 4.8, GPT-5.5, or Gemini 3.1 Pro class) to score outputs against a rubric. Treat the judge the way you treat AI-generated tests: calibrate it before trusting it. Score 20-30 cases by hand, compare with the judge, and check agreement. A judge that rubber-stamps everything is the eval-world equivalent of a tautology test.