9.6Setting SLOs for LLM Features

LLM SLOs should be defined separately from traditional API SLOs:

# llm-slo-definition.yaml
service: ai-chatbot
llm_slos:
  - name: streaming_responsiveness
    metric: time_to_first_token
    target: p95 < 2000ms
    window: 7d

  - name: generation_throughput
    metric: tokens_per_second
    target: avg > 40 tok/s
    window: 7d

  - name: completion_time
    metric: total_generation_time
    target: p95 < 10000ms
    window: 7d

  - name: availability
    metric: successful_requests / total_requests
    target: 99.5%
    window: 30d
    # Note: lower than typical API SLOs because LLM providers
    # have higher baseline error rates

  - name: rate_limit_headroom
    metric: requests_used / rate_limit
    target: peak < 80%
    window: 1d
    alert_at: 70%