Reporting and Observability

Updated Aug 2026

What to Capture

Per-Test Artifacts

Per-Suite Artifacts

A structural advantage of the disk-first CLI: most of this is already on disk. The .playwright-cli/ workspace accumulates snapshots and screenshots as a side effect of the agent working — observability is largely a matter of copying the workspace into your artifacts directory, not instrumenting anything.

Command Logging

Every playwright-cli command the agent executes should be logged:

[2026-07-09T14:23:01Z] playwright-cli open https://app.example.com/login
  → exit 0, 1.2s

[2026-07-09T14:23:02Z] playwright-cli snapshot
  → exit 0, 0.3s, .playwright-cli/snapshot-001.yaml

[2026-07-09T14:23:03Z] playwright-cli fill e3 "user@test.com"
  → exit 0, 0.3s

[2026-07-09T14:23:03Z] playwright-cli fill e4 "***"
  → exit 0, 0.2s

[2026-07-09T14:23:04Z] playwright-cli click e5
  → exit 0, 0.4s

[2026-07-09T14:23:05Z] playwright-cli snapshot
  → exit 0, 0.2s, .playwright-cli/snapshot-002.yaml
  → verified: heading "Welcome, User"

This log enables:

• Replay — run the exact same commands to reproduce an issue
• Timing analysis — identify slow steps
• Debugging — see exactly what the agent did, with each snapshot file as the state evidence for that moment

JSON Report Format

{
  "suite": "authentication",
  "timestamp": "2026-07-09T14:23:00Z",
  "environment": {
    "base_url": "https://staging.example.com",
    "browser": "Chromium (Playwright 1.61)",
    "tool": "playwright-cli",
    "session_mode": "fresh-per-test",
    "headless": true
  },
  "summary": {
    "total": 8,
    "passed": 7,
    "failed": 1,
    "skipped": 0,
    "duration_ms": 18500
  },
  "tests": [
    {
      "name": "login_valid_credentials",
      "status": "pass",
      "duration_ms": 2300,
      "steps": [
        {
          "command": "open https://staging.example.com/login",
          "duration_ms": 1200,
          "exit_code": 0,
          "screenshot": "screenshots/login_valid_step1.png"
        },
        {
          "command": "fill e3 \"test@example.com\"",
          "duration_ms": 300,
          "exit_code": 0
        }
      ]
    },
    {
      "name": "login_expired_account",
      "status": "fail",
      "duration_ms": 5100,
      "error": {
        "message": "Expected text 'Account expired' but got 'Welcome, User'",
        "step": 5,
        "command": "snapshot",
        "screenshot": "failures/login_expired_account/screenshot.png",
        "page_snapshot": "failures/login_expired_account/snapshot.yaml",
        "page_url": "https://staging.example.com/dashboard"
      }
    }
  ]
}

Console Summary Output

Human-readable output for CI logs:

Authentication Test Suite
=========================
Target: https://staging.example.com
Browser: Chromium via playwright-cli (headless, fresh session per test)
Started: 2026-07-09 14:23:00 UTC

  PASS  login_valid_credentials          2.3s
  PASS  login_invalid_password           1.8s
  PASS  login_empty_form                 1.5s
  PASS  login_sql_injection              1.2s
  FAIL  login_expired_account            5.1s
        Expected: "Account expired"
        Got: "Welcome, User"
        Screenshot: failures/login_expired_account/screenshot.png
  PASS  signup_new_user                  3.2s
  PASS  signup_duplicate_email           2.1s
  PASS  logout                           1.3s

=========================
Results: 7 passed, 1 failed (18.5s)

Failure Analysis

When a test fails, the agent (or framework) should capture:

1. Visual State (Screenshot)

playwright-cli screenshot          # PNG in .playwright-cli/ → copy to failures/${TEST_NAME}/

2. Semantic State (YAML Snapshot)

playwright-cli snapshot            # roles, names, refs → copy to failures/${TEST_NAME}/

The snapshot is the artifact the agent re-analyzes: unlike a raw text dump, it preserves structure ("the error alert exists but is empty" vs "the page contains no error region at all" — different bugs).

3. Location (URL)

playwright-cli eval "location.href" > "failures/${TEST_NAME}/current_url.txt"

4. Console Errors (JavaScript)

playwright-cli eval "JSON.stringify(window.__console_errors || [])" > "failures/${TEST_NAME}/console_errors.json"

5. The Trace (When Available)

For Playwright test-runner executions and healer sessions, archive the trace. It contains every action with before/after snapshots, console output, and — since 1.60 — HAR network capture. It is simultaneously the human debugging artifact (Trace Viewer), the agent debugging artifact (feed it back instead of a vague "it failed"), and the compliance artifact ("show me what the AI did" has a literal answer).

6. Agent Reasoning (Optional)

If running with Claude Code, the agent's reasoning about the failure is captured in conversation history. This can be extracted as a markdown file explaining what happened.

Performance Tracking

Per-Command Timing

# Wrapper function that times each command
timed_pw() {
  local start=$(date +%s%N)
  playwright-cli "$@"
  local exit_code=$?
  local end=$(date +%s%N)
  local duration_ms=$(( (end - start) / 1000000 ))
  echo "[timing] playwright-cli $@ → ${duration_ms}ms (exit $exit_code)" >> timing.log
  return $exit_code
}

Trend Analysis

Track test durations over time to catch performance regressions:

{
  "date": "2026-07-09",
  "test": "checkout_flow",
  "duration_ms": 4500,
  "previous_avg_ms": 3200,
  "regression": true,
  "delta_pct": 40.6
}

Alert when a test consistently takes >50% longer than its rolling average.

Integration with CI Dashboards

JUnit XML Output (for Jenkins/GitHub/GitLab)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="authentication" tests="8" failures="1" time="18.5">
    <testcase name="login_valid_credentials" time="2.3"/>
    <testcase name="login_expired_account" time="5.1">
      <failure message="Expected 'Account expired' but got 'Welcome, User'">
        Screenshot: failures/login_expired_account/screenshot.png
        Snapshot: failures/login_expired_account/snapshot.yaml
        URL: https://staging.example.com/dashboard
      </failure>
    </testcase>
  </testsuite>
</testsuites>

Interview Talking Point

"Our reporting captures three things for every test: a command log for reproducibility, screenshots for visual debugging, and structured JSON for programmatic analysis. On failure we also keep the YAML accessibility snapshot and the URL, so the agent can re-analyze the failure semantically — and for runner and healer sessions we archive the Playwright trace, which since 1.60 includes HAR network capture. That trace is the audit trail: when someone asks what the AI actually did in the browser, we replay it. The key insight is that AI-driven tests need richer failure artifacts than traditional tests, because the agent may need to reason about the failure to determine if it's a real bug or a test environment issue — and the disk-first CLI gives us most of those artifacts as a side effect of the agent working."