3.3Statistical Significance for Quality Gates

Quality gate decisions must be statistically rigorous. A naive comparison ("canary error rate is 2.1% vs. control 2.0%") can lead to false conclusions.

Proportions Z-Test for Quality Gates

# quality_gate_statistics.py
import numpy as np
from scipy import stats

def is_canary_safe(control_errors, canary_errors, significance_level=0.05):
    """
    Determine if the canary version is statistically no worse than control.
    Uses a one-tailed proportions z-test.

    Args:
        control_errors: list of 0/1 (0=success, 1=error) for control group
        canary_errors: list of 0/1 for canary group
        significance_level: p-value threshold (default 0.05)

    Returns: (is_safe: bool, p_value: float, details: str)
    """
    n_control = len(control_errors)
    n_canary = len(canary_errors)

    p_control = sum(control_errors) / n_control
    p_canary = sum(canary_errors) / n_canary

    # Pooled proportion under the null hypothesis
    p_pool = (sum(control_errors) + sum(canary_errors)) / (n_control + n_canary)

    # Standard error of the difference
    se = np.sqrt(p_pool * (1 - p_pool) * (1/n_control + 1/n_canary))

    if se == 0:
        return True, 1.0, "No errors in either group"

    # Z-score: is canary WORSE than control?
    z = (p_canary - p_control) / se
    p_value = 1 - stats.norm.cdf(z)  # one-tailed test

    is_safe = p_value > significance_level

    details = (
        f"Control error rate: {p_control:.4%} ({sum(control_errors)}/{n_control})\n"
        f"Canary error rate:  {p_canary:.4%} ({sum(canary_errors)}/{n_canary})\n"
        f"Z-score: {z:.3f}, P-value: {p_value:.4f}\n"
        f"Decision: {'SAFE - no significant degradation' if is_safe else 'UNSAFE - canary is significantly worse'}"
    )

    return is_safe, p_value, details


# Example: control vs canary with identical error rates
control_results = [0]*9800 + [1]*200      # 2.0% error rate (10,000 requests)
canary_results = [0]*980 + [1]*20          # 2.0% error rate (1,000 requests)

safe, p_val, details = is_canary_safe(control_results, canary_results)
print(details)
# Control error rate: 2.0000%
# Canary error rate:  2.0000%
# Z-score: 0.000, P-value: 0.5000
# Decision: SAFE - no significant degradation

Sample Size Requirements

The number of requests needed for a statistically valid comparison depends on the baseline error rate and the minimum detectable effect:

Baseline Error Rate	Minimum Detectable Increase	Required Samples (per group)
0.1%	0.1% (doubling)	~38,000
0.5%	0.25%	~12,000
1.0%	0.5%	~7,000
2.0%	1.0%	~4,000
5.0%	2.5%	~1,500

Practical implication: For services with very low error rates, you need either high traffic volume or longer observation windows to reach statistical significance.