Architecture Decision Records for an AI Test Automation Framework

Updated Aug 2026

ADR-001: Agent-Driven vs Traditional Test Execution

Context

Traditional test frameworks (Selenium, Playwright, Cypress) execute tests as deterministic scripts. AI agents introduce non-deterministic reasoning into the test execution loop.

Decision

Use an agent-as-orchestrator model: the AI agent reads test definitions (natural language or structured), decides how to interact with the application, and reports results. The framework provides the tools (the Playwright CLI), but the agent decides the execution strategy.

Consequences

• (+) Tests are more resilient — the agent can reason about unexpected states
• (+) Test definitions can be higher-level ("verify login works") instead of step-by-step
• (+) Self-healing: when the UI changes, the agent re-snapshots and adapts (and Playwright's healer agent productizes this loop — see the Self-Healing Strategies chapter)
• (-) Non-deterministic — the same test might execute differently each time
• (-) Harder to debug — agent reasoning is opaque compared to line-by-line scripts
• (-) Slower per-test than deterministic execution

Mitigation

• Log every command the agent executes (for reproducibility)
• Screenshot on every state change (for debugging)
• Set deterministic timeouts (prevent infinite loops)
• Archive Playwright traces where available — the flight recorder for what the agent did
• Fallback to deterministic scripts (generated .spec.ts) for critical paths

ADR-002: Playwright CLI Skill as Primary Browser Interface

Context

We evaluated three approaches:

1. Playwright MCP server (structured tools, streamed accessibility trees)
2. Playwright CLI taught via SKILL.md (playwright-cli install --skills)
3. Direct Playwright client library

Decision

Use the Playwright CLI skill as the primary interface, with Playwright MCP reserved for environments without filesystem access.

Rationale

The CLI's disk-first design leaves the bulk of context for reasoning while providing the same accessibility-tree semantics MCP streams — as files under .playwright-cli/ that the agent reads only when needed. This matches Microsoft's own recommendation for coding agents.

Consequences

• (+) ~4x lower token cost than MCP (up to ~10x on long sessions), per Microsoft's benchmark
• (+) Simple setup (npm i -g @playwright/cli + playwright-cli install --skills)
• (+) Composable with other CLI tools; snapshots are greppable YAML
• (+) Same Playwright engine underneath — auto-waiting, actionability checks, cross-browser
• (-) Disk round-trips add latency versus MCP streaming (cheaper in tokens ≠ faster in wall-clock)
• (-) Requires filesystem access — sandboxed surfaces still need MCP
• (-) Refs go stale when the page changes; recovery means a fresh snapshot read

ADR-003: Test Definition Format

Context

Tests need to be defined in a format the agent can read and execute. Options:

1. Natural language descriptions
2. Gherkin (Given/When/Then)
3. Structured YAML/JSON
4. Hybrid (structured with natural language steps)

Decision

Use structured YAML with natural language steps:

test: Login with valid credentials
preconditions:
  - User "test@example.com" exists with password "secret123"
steps:
  - Navigate to the login page
  - Enter email "test@example.com"
  - Enter password "secret123"
  - Click the login button
  - Verify the dashboard loads with welcome message
expected:
  - Dashboard page is displayed
  - Welcome message contains "test@example.com"
hints:  # Optional — the agent resolves concrete refs from snapshots at runtime
  login_page: https://app.example.com/login
  email_field: 'textbox "Email address"'
  password_field: 'textbox "Password"'
  submit_button: 'button "Sign in"'

Rationale

• Natural language steps give the agent freedom to adapt
• Structured format enables programmatic test management
• Hints use accessible role + name (the vocabulary of snapshot YAML), not CSS selectors — the agent maps them to element refs (e3, e5) from a live snapshot, so hints survive markup refactors
• The agent can ignore hints and discover the page from a fresh snapshot if they're stale

This format also plays well with Playwright's first-party workflow: it is deliberately close to the Markdown plans the planner agent writes into specs/ (see the Test Agents chapter), so migrating a suite in either direction is mechanical.

ADR-004: Fresh Sessions for CI, Reused Named Sessions for Development

Context

The Playwright CLI manages browser sessions in the .playwright-cli/ workspace. Named sessions (-s=name) persist across commands; a session can be closed and a new one opened at any time. The classic trade-off — persistent browser for speed vs fresh browser for isolation — maps onto session lifecycle.

Decision

• Development/local: one long-lived named session, reused across runs — no browser startup cost between iterations, and --headed when you want to watch
• CI/CD: a fresh named session per test for isolation, closed when the test ends

Configuration

# Development: reuse one session all afternoon
playwright-cli -s=dev open https://localhost:3000 --headed
# ... iterate ...

# CI: one session per test, torn down after
playwright-cli -s="$TEST_NAME" open "$BASE_URL/login"
# ... test commands with -s="$TEST_NAME" ...
playwright-cli -s="$TEST_NAME" close

Consequences

• (+) Fast feedback during development
• (+) Clean isolation in CI — no state leaks between tests
• (+) Parallel tests come free: each worker uses its own session name
• (-) Reused dev sessions can accumulate state (mitigated by explicit cleanup: state-save/state-load for known-good baselines, or just close and reopen)

ADR-005: Snapshot-and-Screenshot Debugging

Context

When a test fails, the agent needs to understand what went wrong. Options:

1. HTML dump of the page
2. Screenshot capture
3. Accessibility snapshot
4. All of the above

Decision

Capture screenshot + YAML snapshot on every failure. Raw DOM via eval on request.

# On failure, the agent executes:
playwright-cli screenshot     # PNG lands in .playwright-cli/ — copy into failures/
playwright-cli snapshot       # YAML a11y summary — copy into failures/
playwright-cli eval "location.href"   # current URL into the failure record

Rationale

• Screenshots provide visual context for humans reviewing failures
• The YAML snapshot provides semantic context the agent can reason about — roles, names, refs — far more useful than a raw text dump
• Both are files: they cost nothing until someone (human or agent) reads them
• Raw HTML is rarely needed but available via playwright-cli eval "document.documentElement.outerHTML"
• Where a Playwright trace is available (test-runner and healer runs), archive it too — it includes every action, before/after snapshots, console output, and since 1.60 the HAR network capture

ADR-006: Error Recovery Strategy

Context

When a command fails (stale ref, timeout, unexpected state), the agent needs a strategy.

Decision

Three-tier recovery:

Tier 1: Re-snapshot and Re-resolve (Agent Reasoning)

Agent: playwright-cli click e5 → ERROR: ref not found
Agent: "The ref is stale — the page must have changed. Fresh snapshot."
Agent: playwright-cli snapshot
Agent: *reads YAML: button "Submit Form" [ref=e9]*
Agent: playwright-cli click e9 → SUCCESS

Tier 2: Screenshot Analysis

Agent: playwright-cli screenshot
Agent: *reads the PNG*
Agent: "I see a loading spinner. The page hasn't finished loading."
Agent: *waits, re-snapshots — spinner gone, refs present*
Agent: playwright-cli click e9 → SUCCESS

Tier 3: Escalate to the Healer Workflow

Agent: "Recovery inside the run hasn't worked. This is a test-maintenance
problem, not a retry problem."
→ Hand the failing test to Playwright's healer agent: it replays the failure,
  inspects the live UI, patches locators/waits/data and reruns — or concludes
  the functionality itself is broken and reports a bug instead of a green test.

Consequences

• (+) Most failures resolve at Tier 1 (one snapshot read — cheap)
• (+) Tier 2 provides visual debugging even for automated runs
• (+) Tier 3 turns persistent failures into either a durable test fix or a real bug report — never a silently forced pass
• (-) Multi-tier recovery adds latency to failure cases

ADR-007: Test Result Reporting

Context

Test results need to be consumable by:

1. The AI agent (for reasoning about pass/fail)
2. Developers (for debugging)
3. CI systems (for pass/fail gates)

Decision

Three output formats:

Console output (for CI):

PASS  login_valid_credentials      (2.3s)
PASS  login_invalid_password       (1.8s)
FAIL  login_expired_account        (5.1s) — Expected "Account expired", got "Welcome"
PASS  signup_new_user              (3.2s)

4 tests: 3 passed, 1 failed

JSON report (for programmatic consumption):

{
  "suite": "authentication",
  "tests": [
    {
      "name": "login_valid_credentials",
      "status": "pass",
      "duration_ms": 2300,
      "commands": ["open", "snapshot", "fill", "fill", "click", "snapshot"],
      "artifacts": ["login_step1.png", "login_result.yaml"]
    }
  ]
}

Failure artifacts (for debugging):

failures/
├── login_expired_account/
│   ├── screenshot.png
│   ├── snapshot.yaml
│   ├── page_url.txt
│   ├── trace.zip          # when produced by a runner/healer session
│   └── agent_reasoning.md

ADR-008: Parallel Test Execution

Context

Running tests sequentially is slow. But browser tests have shared state issues.

Decision

• Sequential by default in development (one reused session)
• Parallel via named sessions — each worker gets its own isolated session (-s=worker-N), which the CLI keeps separate in the .playwright-cli/ workspace
• Parallel limit: match available CPU cores (each browser instance uses ~200MB RAM + 1 core)

# Parallel execution (4 workers), one named session per test
cat test_list.txt | xargs -P4 -I{} bash -c '
  ./run-single-test.sh "{}"      # script opens/closes session -s="{}"
'

Constraints

• Each parallel worker needs its own browser instance (~200MB RAM)
• 8-core CI machine: max ~6 parallel workers (leave 2 cores for OS + agent)
• Network-bound tests may not benefit from parallelism