9.5Part 4: RAG-Specific Metrics

Retrieval-augmented features fail in two distinct places, so you measure two distinct stages:

Retrieval stage -- did the system fetch the right documents?

• Precision@K: of the top K retrieved chunks, what fraction are actually relevant? Low precision = the generator is fed noise.
• Recall@K: of all relevant chunks in the corpus, what fraction made it into the top K? Low recall = the answer's source material never arrived, and no prompt engineering can fix that.

Generation stage -- did the model use the documents honestly?

• Grounding (faithfulness): is every claim in the answer supported by the retrieved context? This is the anti-hallucination metric.
• Citation accuracy: when the answer cites a source, does that source actually contain the cited claim? Users forgive "I don't know"; they do not forgive a confident answer with a fabricated citation.

The diagnostic value is in the split: a wrong answer with good retrieval metrics is a generation/prompt problem; a wrong answer with bad Recall@K is an indexing/chunking/embedding problem. Without stage-level metrics, teams "fix" retrieval bugs by fiddling with prompts.

@pytest.mark.eval
def test_support_search_retrieval_quality(rag_pipeline, golden_queries):
    """Retrieval-stage gate for the support-articles RAG feature."""
    results = [rag_pipeline.retrieve(q.query, k=5) for q in golden_queries]

    p_at_5 = mean(precision_at_k(r, q.relevant_ids, k=5)
                  for r, q in zip(results, golden_queries))
    r_at_5 = mean(recall_at_k(r, q.relevant_ids, k=5)
                  for r, q in zip(results, golden_queries))

    assert p_at_5 >= 0.80, f"Precision@5 dropped to {p_at_5:.2f}"
    assert r_at_5 >= 0.85, f"Recall@5 dropped to {r_at_5:.2f}"

Run the eval suite whenever any of its three moving parts changes: the prompt, the model version, or the retrieval index. Each is a deployment of behavior, even when no application code changed.