14.4Surface 3: Deterministic Replay of Agent Traces

You cannot debug what you cannot reproduce. The core technique is to record the full trace of an agent run -- every model call, every tool call, every result -- and replay it with the non-deterministic parts stubbed out.

class ReplayHarness:
    """Replay a recorded agent trace with the LLM stubbed out.

    Turns 'the agent did something weird on Tuesday' into a
    deterministic regression test.
    """

    def __init__(self, trace_path: str):
        self.trace = json.load(open(trace_path))

    def replay(self, agent) -> ReplayReport:
        divergences = []
        for step in self.trace["steps"]:
            # Stub the LLM with the recorded decision
            agent.llm = RecordedLLM(step["model_response"])
            actual_tool_call = agent.next_tool_call(step["input_state"])

            if actual_tool_call != step["tool_call"]:
                divergences.append({
                    "step": step["index"],
                    "expected": step["tool_call"],
                    "actual": actual_tool_call,
                })
        return ReplayReport(divergences=divergences)

Two distinct uses, do not conflate them:

1. Harness regression: with the LLM fully stubbed, replay verifies that your code around the model (parsing, tool dispatch, guardrails, state management) still handles the recorded trace identically. Run this in CI on every commit -- it is fast, free, and deterministic.
2. Behavioral comparison: replay the recorded inputs against a live model to see where a new model version or prompt change diverges from the baseline. This is not pass/fail -- it is a diff for a human (or a critic agent) to review before you roll out the change.

This is the same trick as decision caching from the determinism chapter, matured into the standard workflow for agent debugging: every serious agent framework as of July 2026 emits traces precisely so they can be replayed.