14.6Surface 5: Evaluating Non-Determinism with pass^k

A single passing run of an agentic scenario means almost nothing. The standard as of July 2026 is repeat-run evaluation: run each scenario k times and report both the pass rate and pass^k -- whether all k runs passed.

def evaluate_scenario(scenario, k: int = 10) -> ScenarioReport:
    results = [run_in_sandbox(scenario) for _ in range(k)]
    passes = sum(1 for r in results if r.passed)
    return ScenarioReport(
        scenario=scenario.name,
        pass_rate=passes / k,          # capability signal
        pass_all_k=(passes == k),      # reliability signal
        failure_modes=cluster_failures([r for r in results if not r.passed]),
    )

The two numbers answer different questions:

• pass rate (capability): can the agent do this at all?
• pass^k (reliability): can you depend on it?

The gap between them is brutal at realistic reliability levels. A per-run pass rate of 90% collapses to a 35% chance of 10 consecutive passes (0.9^10); even 99% per-run gives only ~90% for pass^10. Interviewers probe exactly this: an agent that "works in the demo" and an agent you can put in front of customers are separated by an order of magnitude of reliability engineering.

Practical policy: gate releases on pass^k for the scenarios that matter (k sized to your risk tolerance), track pass rate over time to catch capability regressions, and always cluster the failures -- three failures with one root cause is a bug; three unrelated failures is a reliability profile.