13.3Cost Optimization Strategies

Strategy 1: Model Tiering

Use cheaper models for simple decisions, expensive models for complex reasoning.

class TieredModelRouter:
    """Route agent decisions to appropriate model tier."""

    def __init__(self):
        # Pin exact model versions in production config, and revisit them
        # quarterly -- as of July 2026 the frontier tier is Claude Opus 4.8.
        self.fast_model = "claude-haiku-latest"       # Cheap, fast
        self.standard_model = "claude-sonnet-latest"  # Balanced
        self.premium_model = "claude-opus-latest"     # Best reasoning

    def get_model(self, decision_type: str) -> str:
        # Simple navigation/clicking decisions → cheap model
        if decision_type in ["navigate", "click", "type"]:
            return self.fast_model

        # Test assertion evaluation → standard model
        if decision_type == "evaluate":
            return self.standard_model

        # Bug analysis, complex reasoning → premium model
        if decision_type in ["diagnose_bug", "generate_test_plan"]:
            return self.premium_model

        return self.standard_model  # Default

Cost impact: Model tiering typically reduces costs by 40-60% with minimal quality loss. Most agent steps are simple navigation decisions that a smaller model handles just as well.

Strategy 2: Decision Caching

If the page has not changed, the agent's decision should not change either.

class CachedDecisionMaker:
    def __init__(self, llm, cache_ttl=300):
        self.llm = llm
        self.cache = {}
        self.cache_ttl = cache_ttl
        self.cache_hits = 0
        self.cache_misses = 0

    def decide(self, observation_hash: str, prompt: str) -> str:
        # Check cache
        if observation_hash in self.cache:
            entry = self.cache[observation_hash]
            if time.time() - entry["timestamp"] < self.cache_ttl:
                self.cache_hits += 1
                return entry["decision"]

        # Cache miss: call LLM
        self.cache_misses += 1
        decision = self.llm.generate(prompt)
        self.cache[observation_hash] = {
            "decision": decision,
            "timestamp": time.time()
        }
        return decision

    @property
    def hit_rate(self) -> float:
        total = self.cache_hits + self.cache_misses
        return self.cache_hits / total if total > 0 else 0

Cost impact: Caching reduces LLM calls by 20-40% in test suites with repeated page states (e.g., multiple tests starting from the same login page).

Strategy 3: Progressive Testing

Run the full agent suite nightly. Run a constrained smoke subset on every commit.

# CI configuration
if event_type == "push":
    # Every commit: run only smoke tests with tight budgets
    config = HarnessConfig(max_steps=10, max_tokens=10_000)
    run_tests(SMOKE_TESTS, config)

elif event_type == "nightly":
    # Nightly: run full suite with generous budgets
    config = HarnessConfig(max_steps=30, max_tokens=50_000)
    run_tests(ALL_TESTS, config)

elif event_type == "weekly":
    # Weekly: run exploratory agents with no budget limits
    config = HarnessConfig(max_steps=100, max_tokens=200_000)
    run_tests(EXPLORATORY_TESTS, config)

Cost impact: Reduces daily CI cost by 60-80% while maintaining nightly coverage.

Strategy 4: Early Exit on Critical Failures

If the login page is broken, do not spend tokens testing the dashboard.

class SmartTestRunner:
    def run_with_early_exit(self, tests: list, config):
        # Run critical path tests first
        critical = [t for t in tests if t.priority == "critical"]
        standard = [t for t in tests if t.priority == "standard"]

        for test in critical:
            result = self.run_test(test, config)
            if result.status == "fail":
                # Critical test failed: skip standard tests
                return SuiteResult(
                    status="ABORT",
                    reason=f"Critical test failed: {test.name}",
                    skipped=standard,
                    tokens_saved=self.estimate_tokens(standard)
                )

        # Critical tests passed: run standard tests
        for test in standard:
            self.run_test(test, config)

Cost impact: Saves 50-90% of tokens when critical flows are broken.