6.3Canary Deployment

How It Works

The new version is deployed to a small subset of servers (the "canary"). A small percentage of real traffic (typically 1-5%) is routed to the canary. If metrics look good, traffic gradually increases until the canary becomes the new production.

                    Load Balancer
                   /      |      \
              v2.3     v2.3     v2.4 (canary)
              95%                5% of traffic
                                ↑
                           Monitor metrics here:
                           - Error rate
                           - Latency p99
                           - Business metrics

Where Tests Run

1. Before canary deployment: Run the standard test suite against a staging environment
2. During canary: Rely on synthetic monitoring and real-time metrics rather than full test suites -- the canary is serving real traffic
3. Promotion decision: Based on metric comparison between canary and baseline

# Example: Canary monitoring
canary-monitoring:
  runs-on: ubuntu-latest
  steps:
    - name: Wait for canary stabilization
      run: sleep 300  # 5 minutes for metrics to accumulate

    - name: Compare canary metrics
      run: |
        CANARY_ERROR_RATE=$(curl -s "$METRICS_API/canary/error-rate")
        BASELINE_ERROR_RATE=$(curl -s "$METRICS_API/baseline/error-rate")

        if (( $(echo "$CANARY_ERROR_RATE > $BASELINE_ERROR_RATE * 1.1" | bc -l) )); then
          echo "Canary error rate ($CANARY_ERROR_RATE) exceeds baseline ($BASELINE_ERROR_RATE) by >10%"
          exit 1
        fi

    - name: Run synthetic smoke tests
      run: npx playwright test --project=smoke
      env:
        BASE_URL: https://canary.example.com

Risk Level: Medium

Real users hit the new code early, but only a small percentage. If the canary is bad, only 5% of users are affected, and rollback is automatic.

QA Implications

• Synthetic monitoring tests must be lightweight and fast (they run against live traffic)
• Focus on metrics: error rates, latency percentiles, business conversion rates
• You need good observability (logging, metrics, tracing) to detect canary issues
• Test the rollback mechanism itself -- a canary that cannot roll back is useless