Self-Healing Strategies for Agent-Driven Tests

Updated Aug 2026

What "Self-Healing" Means

Traditional self-healing (testRigor, mabl, etc.): When a selector breaks, the tool automatically finds an alternative using heuristics (visual position, text content, nearby elements).

Agent-driven self-healing: When a step breaks, the AI agent reasons about the failure and decides how to recover. This is fundamentally more flexible because the agent can understand context, not just apply rules.

As of 2026 this argument has a productized reference implementation: Playwright's healer agent (first-party since v1.56). Its loop — replay the failing test, inspect the live UI, patch locators/waits/data, rerun until green or conclude the functionality itself is broken — is exactly the agent-reasoning approach this chapter builds, shipped by the framework vendor. Keep it in mind as the standard throughout.

The Three Levels of Recovery

Level 1: Re-snapshot and Re-resolve

With ref-driven interaction, the classic "selector broke" failure becomes "ref went stale." The recovery is a fresh snapshot:

Agent: Bash("playwright-cli click e5")
→ Error: ref e5 not found on current page

Agent thinking: "The ref is stale — the page changed since my last snapshot.
Let me look at what's actually there now."

Agent: Bash("playwright-cli snapshot")
→ .playwright-cli/snapshot-014.yaml
Agent reads:
  - button "Cancel" [ref=e7]
  - button "Submit Form" [ref=e8]
  - button "Save Draft" [ref=e9]

Agent thinking: "The submit button is now 'Submit Form' at ref e8.
Same accessible role, same intent."

Agent: Bash("playwright-cli click e8")
→ Success

Token cost: one snapshot read + reasoning (~1,500-5,000 tokens depending on page size) Success rate: high for renames, reordering, and markup refactors — the accessibility tree changes far less often than the DOM

Level 2: Page Analysis with Screenshot + Snapshot

When Level 1 doesn't work, the agent takes a deeper look:

Agent: Bash("playwright-cli snapshot")
→ YAML shows almost nothing: text "Loading… Please wait"

Agent: Bash("playwright-cli screenshot")
Agent: *reads the PNG*

Agent thinking: "There's a loading spinner. The page never finished loading —
this isn't a locator problem at all."

Agent: *waits, re-snapshots — content present now*
Agent: Bash("playwright-cli click e8")
→ Success

Token cost: ~1 screenshot read + 1-2 snapshot reads Success rate: catches timing issues, loading states, redirects, blocking modals

Level 3: Hand the Test to the Healer Agent

When failures persist across runs, stop patching inside the run — this is now test maintenance, and it's what Playwright's healer agent is for:

CI: add-valid-todo.spec.ts FAILED

Healer: *runs the failing test, replays the failing step*
Healer: *inspects the live UI in the actual failing session
        (browser.bind() means it debugs the real state, not a reproduction)*
Healer: *patches the locator, adjusts a wait, fixes test data*
Healer: *reruns until green*

— or —

Healer: "The 'Save' action returns a 500. The functionality is broken.
This test should stay red."

That second outcome is the whole point. A healer that always makes the test pass is a bug-hiding machine. Playwright's framing — "the app is actually broken" is a valid terminal state — is the honesty standard you should hold any self-healing tool to in an evaluation, commercial vendors included.

Cost: a full agent session — expensive, but it produces a durable fix (a patched test in a PR) rather than an in-run workaround Success rate: handles major redesigns; and when it "fails," it has found you a real bug

Recovery Strategy Matrix

Implementing Self-Healing in Your Framework

Wrapper Function (Scripted Tier-1 Healing)

For scripted (non-agent) runs, a thin healing wrapper can resolve an element by accessible name from a fresh snapshot:

# smart_click: click by expected accessible name, resolving the ref at runtime
smart_click() {
  local expected_name="$1"     # e.g. "Submit"

  # Take a fresh snapshot; the CLI returns the YAML file path
  local snap
  snap=$(playwright-cli snapshot)

  # Find a button/link whose accessible name matches, extract its ref
  local ref
  ref=$(grep -iE "(button|link) \"[^\"]*${expected_name}[^\"]*\"" "$snap" \
        | grep -oE 'ref=e[0-9]+' | head -1 | cut -d= -f2)

  if [ -n "$ref" ] && playwright-cli click "$ref"; then
    echo "[heal] clicked ${expected_name} via ref ${ref}"
    return 0
  fi

  # Evidence for the humans (and the healer)
  playwright-cli screenshot
  echo "[heal] could not resolve an element named: $expected_name"
  return 1
}

# Usage:
smart_click "Submit"
smart_click "Log in"

Notice what this is: a five-line, grep-based rediscovery of the element by its accessible name — because the snapshot is plain YAML on disk, tier-1 healing doesn't even need an LLM.

Agent-Native Approach

Rather than scripting recovery logic, let the agent handle it naturally:

# Test definition with recovery hints
test: Submit contact form
steps:
  - Fill in the contact form
  - Click the submit button
recovery_hints:
  submit_button:
    accessible_name: "Submit"
    fallback_names: ["Send", "Send message"]
    semantic: "The button that submits the form"

The agent resolves refs from the live snapshot using the hints — and because hints are expressed as accessible names rather than CSS selectors, they survive markup refactors.

Self-Healing vs Flaky Tests

What's the Difference?

Flaky test: The test itself is unreliable (race conditions, timing issues, external dependencies). Self-healing doesn't fix this — it masks it.

Broken locator: The application changed, the test's element resolution is stale. Self-healing legitimately fixes this.

How to Tell Them Apart

The Rule

Self-healing should fix broken element resolution, not mask flaky tests. If a test needs healing on every run, it's flaky and needs to be fixed at the source. (And if the healer keeps concluding "the app is broken," believe it — that's not a healing failure, that's a bug report.)

Tracking Self-Healing Events

Log every healing event for analysis:

{
  "timestamp": "2026-07-09T14:23:05Z",
  "test": "login_valid",
  "step": 4,
  "expected_element": "button \"Sign in\"",
  "stale_ref": "e5",
  "healed_ref": "e8",
  "healing_level": 1,
  "reason": "Button renamed from 'Submit' to 'Sign in' in UI refresh"
}

Review healing logs weekly:

• High healing rate on one test → Regenerate or hand-fix that test
• High healing rate across tests → Major UI refactor happened; rerun the planner/generator over the affected area instead of healing test-by-test
• Healing always escalating to Level 3 → Your test flows are too coupled to page structure; push for accessible names and test IDs

Locator Robustness Hierarchy

The agent interacts via snapshot refs at runtime, but the tests the generator writes (and the patches the healer makes) are .spec.ts files with real locators — so locator robustness still matters. From most to least robust:

Recommendation for AI-driven tests: role + accessible name first — it's the same vocabulary the snapshots and the agents speak, so healed tests stay consistent with how they were explored. Use data-testid where accessible names are unstable (requires collaboration with developers).

Interview Talking Point

"Our self-healing has three tiers. First, stale refs: the agent re-snapshots — the snapshot is a YAML accessibility summary on disk — and re-resolves the element by role and accessible name; that handles most renames and refactors for the cost of one snapshot read. Second, state analysis: screenshot plus snapshot to catch loading states, redirects, and blocking modals. Third, persistent failures go to Playwright's healer agent, which replays the failure, inspects the live UI, and patches the test — or concludes the app itself is broken and leaves the test red. That last part is the standard I hold any self-healing tool to: a healer that always forces green is a bug-hiding machine. And we track healing events — a test that needs healing every run isn't being healed, it's flaky, and it gets fixed at the source."