19.9"Deterministic vs Non-Deterministic Testing"

What they say: "How do you handle the non-deterministic nature of AI?"

What they mean: Traditional tests produce the same result every time (deterministic). AI agents might take different paths (non-deterministic).

What you should say: "We achieve 'practical determinism' — the agent follows the same path the vast majority of the time because the SKILL.md instructions and test plans constrain its decisions. The divergence happens during error recovery, where non-determinism is actually a feature — the agent tries approaches a deterministic script can't.

We mitigate the risk through:

1. Command logging — see exactly what the agent did
2. Playwright traces — replayable evidence of each state, network included
3. CI gates — pass/fail is deterministic even if the path varies
4. Repeat-run measurement — for agentic flows we run the same scenario k times and track pass^k-style stability, not just single-run pass"