LLM Evals and AI Test-Suite Quality

Updated Jul 2026

Two Evaluation Problems, One Skill Set

By mid-2026, QA job postings routinely list LLM eval design as a core competency, alongside RAG metrics and frameworks like Ragas, TruLens, and OpenAI Evals. This is not a fad -- it reflects two evaluation problems that now land on the QA engineer's desk:

1. Evaluating the tests AI wrote for you. If AI generates half your suite, "how good is the suite?" needs a quantitative answer, not a gut feeling.
2. Evaluating the AI inside your product. When the feature under test is itself an LLM (a chatbot, a summarizer, a RAG-backed search), classical pass/fail assertions stop working -- and someone has to design the evaluation harness.

Both problems share the same discipline: define quality metrics, build a measurement harness, set thresholds, and track trends. That is test engineering. You already do this -- you are just pointing it at new targets.

Part 1: Measuring AI Test-Suite Quality

The review checklist and quality gates from the previous chapters are per-test judgments. At the suite level, three metrics tell you objectively whether AI generation is helping or quietly degrading your safety net:

Why these three together: coverage tells you what code the tests touch, mutation score tells you whether the assertions would notice a bug there, and flake rate tells you whether the signal can be trusted run-to-run. A suite can look great on any one of them and still be rotten -- 90% coverage made of tautology tests has a dismal mutation score; a high-mutation-score suite that flakes 5% of the time trains the team to ignore red builds.

Track the Deltas per Generation Session

The practical trick is to measure per AI-generation session, not just globally:

# Before the session: snapshot the baseline
pytest --cov=app --cov-report=json:cov_before.json

# ... generate, curate, integrate tests (see the curation workflow) ...

# After the session: compare
pytest --cov=app --cov-report=json:cov_after.json
python scripts/coverage_delta.py cov_before.json cov_after.json
# Output: "+4.2 percentage points, 312 newly covered lines, 0 lines lost"

Do the same with mutation testing scoped to the new tests only (mutation testing the whole suite on every PR is too slow -- run it nightly, scoped to modules the AI touched). If a generation session adds 30 tests, +0.5% coverage, and no mutation-score improvement, the session produced bulk, not protection -- fix the prompt template, not just the tests.

Part 2: Prompts Are Code -- Test Them Like Code

Your prompt templates determine the quality of every suite the team generates. A silent regression in a template (or a model upgrade that changes behavior under the same template) degrades every future generation session. The fix is the same discipline you apply to source code:

• Version templates in the repo, change them via PR (covered in the prompt templates chapter).
• Review prompt changes as behavioral changes -- a template edit that drops the "include negative cases" instruction is a coverage regression waiting to happen.
• Test templates against golden inputs: a fixed spec excerpt with known expected properties of the output.

A prompt regression test does not assert exact output (LLM output varies); it asserts measurable properties of the output:

def test_api_template_generates_negative_cases(llm, golden_spec):
    """The api-schema-to-tests template must produce error-path tests."""
    suite = llm.generate(template="api-schema-to-tests", artifact=golden_spec)

    tests = parse_test_functions(suite)
    negative = [t for t in tests if asserts_error_status(t)]

    assert len(tests) >= 15, "Template output volume dropped"
    assert len(negative) / len(tests) >= 0.35, "Happy-path bias regression"
    assert all(has_assertions(t) for t in tests), "Assertion-free tests appeared"

Run this on every template change and after every model version bump. This is your first taste of eval thinking: probabilistic output, property-based assertions, thresholds instead of exact matches.

Part 3: LLM Evals -- Testing the AI Inside Your Product

When the product embeds an LLM, assert response == expected is dead on arrival: the same input can produce different valid outputs. The industry answer is evals -- test suites where each case is scored (by metrics, heuristics, or a judge model) and the suite passes on aggregate thresholds.

The Mental Model: Evals Are Parametrized Tests with Scored Assertions

The Frameworks to Know

You do not need deep expertise in all three. You need to be able to say what each is for, and to design an eval suite in one of them: pick the metric, build the golden dataset (20-100 curated cases beats 10,000 scraped ones), set the threshold, wire it into CI so a prompt change that drops faithfulness below threshold blocks the merge -- exactly like a coverage gate.

LLM-as-Judge, Reviewed Like AI Tests

Model-graded evals use a strong model (as of July 2026: Claude Opus 4.8, GPT-5.5, or Gemini 3.1 Pro class) to score outputs against a rubric. Treat the judge the way you treat AI-generated tests: calibrate it before trusting it. Score 20-30 cases by hand, compare with the judge, and check agreement. A judge that rubber-stamps everything is the eval-world equivalent of a tautology test.

Part 4: RAG-Specific Metrics

Retrieval-augmented features fail in two distinct places, so you measure two distinct stages:

Retrieval stage -- did the system fetch the right documents?

• Precision@K: of the top K retrieved chunks, what fraction are actually relevant? Low precision = the generator is fed noise.
• Recall@K: of all relevant chunks in the corpus, what fraction made it into the top K? Low recall = the answer's source material never arrived, and no prompt engineering can fix that.

Generation stage -- did the model use the documents honestly?

• Grounding (faithfulness): is every claim in the answer supported by the retrieved context? This is the anti-hallucination metric.
• Citation accuracy: when the answer cites a source, does that source actually contain the cited claim? Users forgive "I don't know"; they do not forgive a confident answer with a fabricated citation.

The diagnostic value is in the split: a wrong answer with good retrieval metrics is a generation/prompt problem; a wrong answer with bad Recall@K is an indexing/chunking/embedding problem. Without stage-level metrics, teams "fix" retrieval bugs by fiddling with prompts.

@pytest.mark.eval
def test_support_search_retrieval_quality(rag_pipeline, golden_queries):
    """Retrieval-stage gate for the support-articles RAG feature."""
    results = [rag_pipeline.retrieve(q.query, k=5) for q in golden_queries]

    p_at_5 = mean(precision_at_k(r, q.relevant_ids, k=5)
                  for r, q in zip(results, golden_queries))
    r_at_5 = mean(recall_at_k(r, q.relevant_ids, k=5)
                  for r, q in zip(results, golden_queries))

    assert p_at_5 >= 0.80, f"Precision@5 dropped to {p_at_5:.2f}"
    assert r_at_5 >= 0.85, f"Recall@5 dropped to {r_at_5:.2f}"

Run the eval suite whenever any of its three moving parts changes: the prompt, the model version, or the retrieval index. Each is a deployment of behavior, even when no application code changed.

Interview Talking Point

"I treat evaluation as one skill applied to two targets. For AI-generated test suites, I track mutation score, coverage delta per generation session, and flake rate -- coverage tells me what the tests touch, mutation score whether they'd catch a bug there, flake rate whether I can trust the signal. For LLM-backed features, I design evals instead of exact-match assertions: a golden dataset, scored metrics with thresholds, run in CI on every prompt or model change. For RAG features I measure the stages separately -- Precision@K and Recall@K for retrieval, grounding and citation accuracy for generation -- because the fix for a retrieval failure is completely different from the fix for a hallucination. And I version and regression-test the prompts themselves, because a prompt change is a behavior change."

Key Takeaway

Evaluation is the through-line of AI-era QA. Measure AI-generated suites with mutation score, coverage deltas, and flake rate so "the AI wrote lots of tests" never substitutes for "the suite catches bugs." Treat prompts as code: versioned, reviewed, and regression-tested against golden inputs. And when the product itself contains an LLM, bring eval frameworks (Ragas, TruLens, OpenAI Evals) and stage-level RAG metrics (Precision@K, Recall@K, grounding, citation accuracy) -- the QA engineer who can design these evals is doing the same job as always: turning "it seems fine" into a measured, gated, repeatable answer.