11.4Making Agents Deterministic Enough for CI

If you want to use agents in CI pipelines where non-determinism breaks builds, apply these techniques:

Technique 1: Fix the Seed

# Deterministic model output via temperature=0 and fixed seed
response = client.chat.completions.create(
    model="gpt-5.5",
    messages=messages,
    temperature=0,
    seed=42  # Fixed seed for reproducibility
)

Limitation: Even with temperature=0 and a fixed seed, LLM outputs are not guaranteed to be identical across API versions or model updates.

Technique 2: Cache Agent Decisions

class DeterministicAgent:
    """Replays cached decisions for reproducibility."""
    def __init__(self, cache_path: str):
        self.cache = self.load_cache(cache_path)
        self.step = 0

    def next_action(self):
        if self.step in self.cache:
            action = self.cache[self.step]  # Replay cached decision
        else:
            action = self.llm.decide()       # Generate new decision
            self.cache[self.step] = action   # Record for next run
        self.step += 1
        return action

    def save_cache(self):
        """Save decisions for future deterministic replays."""
        with open(self.cache_path, "w") as f:
            json.dump(self.cache, f)

How it works: On the first run, the agent generates decisions and caches them. On subsequent runs, it replays the cached decisions exactly. If the page changes and a cached decision becomes invalid, the cache is invalidated and a new decision is generated.

Technique 3: Assert on Outcomes, Not Paths

# BAD: asserts the exact sequence of agent actions
assert agent.history == [
    "navigate /login",
    "type email test@test.com",
    "type password secret",
    "click submit"
]

# GOOD: asserts the final state regardless of how the agent got there
assert agent.browser.url == "https://app.example.com/dashboard"
assert "Welcome" in agent.browser.text("h1")

This is the most practical technique. The agent may take a different path on each run, but the test passes as long as the final state is correct.

Technique 4: Measure Stability with Repeat Runs (pass^k)

Before promoting an agentic test into any pipeline, quantify its non-determinism instead of guessing:

def pass_at_all_k(test, k: int = 5) -> float:
    """Run the same test k times; it is stable only if ALL k runs pass."""
    results = [test.run() for _ in range(k)]
    return all(r.status == "pass" for r in results)

A test that passes 95% of individual runs sounds solid -- but the probability that it passes 5 consecutive runs is only 0.95^5 ≈ 77%. This "pass^k" style of repeat-run evaluation is the standard way to measure agent reliability as of July 2026, and it is covered in depth in Testing Agentic Systems.