2.3Canary Analysis with Kayenta (Spinnaker)

Kayenta is Netflix's automated canary analysis tool, integrated with Spinnaker. It compares metrics between the canary and baseline versions and produces a statistical judgment.

{
  "canaryConfig": {
    "name": "checkout-service-canary",
    "judge": {
      "judgeConfigurations": {},
      "name": "NetflixACAJudge-v1.0"
    },
    "metrics": [
      {
        "name": "error_rate",
        "query": {
          "type": "prometheus",
          "customInlineTemplate": "sum(rate(http_requests_total{status=~\"5..\",app=\"checkout\",version=\"${scope}\"}[5m])) / sum(rate(http_requests_total{app=\"checkout\",version=\"${scope}\"}[5m]))"
        },
        "analysisConfigurations": {
          "canary": {
            "direction": "increase",
            "critical": true,
            "mustHaveData": true
          }
        },
        "scopeName": "default"
      },
      {
        "name": "latency_p99",
        "query": {
          "type": "prometheus",
          "customInlineTemplate": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{app=\"checkout\",version=\"${scope}\"}[5m])) by (le))"
        },
        "analysisConfigurations": {
          "canary": {
            "direction": "increase",
            "critical": true
          }
        },
        "scopeName": "default"
      },
      {
        "name": "saturation_cpu",
        "query": {
          "type": "prometheus",
          "customInlineTemplate": "avg(rate(container_cpu_usage_seconds_total{app=\"checkout\",version=\"${scope}\"}[5m]))"
        },
        "analysisConfigurations": {
          "canary": {
            "direction": "increase",
            "critical": false
          }
        },
        "scopeName": "default"
      }
    ],
    "classifier": {
      "groupWeights": {
        "Errors": 40,
        "Latency": 35,
        "Saturation": 25
      }
    }
  }
}

How Kayenta Scores Canaries

1. Collect metrics from both canary and baseline for the analysis window
2. Compare distributions using the Mann-Whitney U test (non-parametric)
3. Score each metric as Pass, Marginal, or Fail
4. Apply group weights (Errors 40%, Latency 35%, Saturation 25%)
5. Produce a final score (0-100). Typically, >70 = promote, <50 = rollback, 50-70 = extend observation