10.3Cold Start Testing Pattern

Serverless LLM deployments (AWS Bedrock, self-hosted on Lambda/Cloud Run) exhibit cold start penalties that can dramatically affect user experience:

// k6-cold-start-test.js
import http from 'k6/http';
import { Trend } from 'k6/metrics';
import { sleep } from 'k6';

const coldStartLatency = new Trend('cold_start_latency', true);
const warmLatency = new Trend('warm_latency', true);

export const options = {
  iterations: 10,
  vus: 1, // sequential to isolate cold starts
};

export default function () {
  // Cold start: wait long enough for the instance to scale down
  sleep(300); // 5 minutes idle -- adjust based on your provider's scale-down policy

  const coldRes = http.post('https://llm.example.com/v1/completions', JSON.stringify({
    prompt: "Hello", max_tokens: 5,
  }), { headers: { 'Content-Type': 'application/json' }, timeout: '60s' });

  coldStartLatency.add(coldRes.timings.duration);
  console.log(`Cold start: ${coldRes.timings.duration}ms`);

  // Warm requests: rapid fire while the instance is hot
  for (let i = 0; i < 5; i++) {
    const warmRes = http.post('https://llm.example.com/v1/completions', JSON.stringify({
      prompt: "Hello", max_tokens: 5,
    }), { headers: { 'Content-Type': 'application/json' }, timeout: '30s' });

    warmLatency.add(warmRes.timings.duration);
    sleep(1);
  }
}

Cold Start Mitigation Strategies

Strategy	How It Works	Trade-off
Provisioned concurrency	Pre-warm N instances (AWS Lambda, Cloud Run min-instances)	Cost: you pay for idle capacity
Keep-alive pings	Periodic health check requests prevent scale-to-zero	Minimal cost, adds complexity
Model caching	Keep model weights in memory across invocations	Requires persistent runtime (not pure serverless)
Edge deployment	Deploy smaller models at the edge for low-latency inference	Limited model capability