9.7Benchmarking Across Providers

When evaluating LLM providers, run standardized benchmarks:

# llm_benchmark.py
BENCHMARK_PROMPTS = [
    {"name": "short_qa", "prompt": "What is 2+2?", "expected_tokens": 10},
    {"name": "medium_summary", "prompt": "Summarize the key differences between REST and GraphQL in 3 sentences.", "expected_tokens": 80},
    {"name": "long_generation", "prompt": "Write a Python function that implements binary search with detailed docstring.", "expected_tokens": 200},
]

def benchmark_provider(client, model: str, runs: int = 10) -> dict:
    """Benchmark a provider/model combination."""
    results = {}
    for prompt_config in BENCHMARK_PROMPTS:
        metrics = []
        for _ in range(runs):
            m = measure_streaming(client, prompt_config["prompt"], model)
            metrics.append(m)

        results[prompt_config["name"]] = {
            "ttft_p50": sorted([m.ttft_ms for m in metrics])[len(metrics)//2],
            "ttft_p95": sorted([m.ttft_ms for m in metrics])[int(len(metrics)*0.95)],
            "tps_avg": sum(m.tokens_per_second for m in metrics) / len(metrics),
            "total_p50": sorted([m.total_generation_ms for m in metrics])[len(metrics)//2],
        }
    return results

This data informs provider selection, fallback prioritization, and SLO calibration. Run benchmarks weekly to track provider performance trends.