9.5Context Window Economics

The size of the prompt directly affects latency and cost. Understanding this relationship is critical for performance optimization:

Context Usage	Typical TTFT Impact	Cost Impact	Optimization
< 1K tokens	Baseline	Baseline	None needed
1-4K tokens	+100-300ms	2-4x	Summarize context
4-16K tokens	+300ms-1s	4-16x	RAG with relevance filtering
16-64K tokens	+1-5s	16-64x	Aggressive context pruning
64-128K tokens	+5-15s	64-128x	Redesign the approach

Practical optimization strategies:

1. Prompt caching. Many providers cache the prefix of repeated prompts, reducing TTFT for subsequent requests with the same system prompt.
2. Context pruning. Remove irrelevant conversation history before sending to the model.
3. Semantic caching. Cache responses for semantically similar queries to avoid LLM calls entirely.
4. Model routing. Send simple queries to smaller, faster models. Reserve large models for complex tasks.