17.4Category 4: Practical Scenarios

Q15: "Walk me through debugging a failing browser test."

Answer: "The failure artifacts give us everything:

1. Command log — I see the exact sequence of CLI commands. Step 7 was playwright-cli click e17 which timed out.

2. Trace — I open the Playwright trace: every action with before/after snapshots, console output, and network capture. I can see a modal overlay appeared right before the click.

3. Snapshot + screenshot — The saved YAML snapshot shows a dialog node ('Are you sure you want to proceed?') that the test didn't expect; the screenshot confirms it visually.

4. Fix — I add a step to handle the confirmation dialog before the original click, or update the spec to expect it.

If the healer agent ran, it might have already patched this — but we still review its diff to decide whether the plan should change. Healer patches are code review items, not silent fixes."

Q16: "How would you test a Single Page Application?"

Answer: "SPAs have specific challenges: navigation doesn't trigger full page loads, content renders asynchronously, and URLs may not change.

Key techniques:

1. Trust actionability, not page load events. The engine's auto-wait means a click on a ref proceeds only when the element is actually interactable — that absorbs most SPA timing issues without explicit waits.

2. Snapshot after state changes:

playwright-cli click e12          # client-side route change
playwright-cli snapshot           # fresh snapshot — new refs for the new view

Element refs are per-snapshot; after a route transition you re-snapshot rather than reuse stale refs — which is exactly the discipline SPAs require anyway.

3. Verify content, not URL. Client-side routing may not change the URL meaningfully; assert on what the snapshot shows (headings, roles, text), not the address bar.

4. Handle lazy loading by scrolling and re-snapshotting to confirm the lazily rendered components exist."

Q17: "How do you handle authentication across tests?"

Answer: "Three strategies:

Strategy 1 — Login via UI (realistic, slow): Drive the actual login form once: open, snapshot, fill email/password refs, press Enter, verify the dashboard appears.

Strategy 2 — Saved state (fast, reliable):

# Once, after a real login:
playwright-cli state-save auth-state

# Every other test:
playwright-cli state-load auth-state
playwright-cli goto https://app.example.com/dashboard   # already authenticated

Strategy 3 — API auth (for CI): Fetch a token via the API, inject it with playwright-cli eval into localStorage or cookies, then navigate.

We use Strategy 1 for the actual login test and Strategy 2 for everything else — one thorough login verification plus fast, deterministic setup for the rest. This is also the seed-test pattern Playwright's test agents formalize: seed.spec.ts provides the authenticated context every agent inherits."

Q18: "How do you handle tests that interact with third-party services?"

Answer: "Three approaches:

1. Mock at the network level: intercept fetch() via playwright-cli eval (or route interception in the underlying framework) so calls to the third party return canned responses.

2. Use test/sandbox environments: Most payment providers (Stripe, PayPal) have sandbox modes. Configure the app to use sandbox credentials in the test environment.

3. Stop at the boundary: Test up to the point of third-party interaction, verify the outgoing request payload (the trace's network capture shows it), then skip the actual external call."

Q19: "What metrics do you track for your test automation?"

Answer: "Five key metrics:

1. Test reliability rate — % of tests that pass consistently (target: >98%)
2. Self-healing rate — % of runs where the agent recovered from a failure (track over time — should decrease as the suite stabilizes; every healer patch is reviewed)
3. Execution time — Per-test and per-suite (detect performance regressions)
4. Token cost — Per-test and per-suite (budget management; the reason we're on the CLI path)
5. Failure-to-fix time — How long between a test failure and the fix being merged (measures the value of failure artifacts)

We track these in a JSON report per run and graph trends weekly. A spike in healing rate means a UI deployment changed something. A spike in execution time means either the app got slower or a test got stuck."

Q20: "Where do you see AI-driven testing going in the next 2-3 years?"

Answer: "Three directions:

1. From assistants to first-party agents — already happened. Playwright ships planner/generator/healer agents today; the debate moved from 'should AI write tests?' to 'how do we review what it writes?' Expect Cypress (cy.prompt()) and the commercial platforms to keep converging on the same plan → generate → heal loop.

2. Persistent application memory. Today's agents rediscover the app every session. The next frontier is an 'app map' the agent maintains across sessions — Vibium's planned Cortex (SQLite + embeddings) is the clearest articulation, but whoever ships it changes the economics of exploratory agent testing. Watch this space.

3. Autonomous QA at the suite level. Agents that don't just execute predefined tests but explore, identify risks, and grow the suite — OpenObserve's 'Council of Sub Agents' (8 specialized agents growing a suite from 380 to 700+ tests) is an early production example.

The agent skills pattern — lightweight markdown teaching agents domain knowledge — has become the standard interface, portable across Claude Code, Cursor, Gemini CLI, and Codex CLI. Browser automation is just one domain; the same pattern applies to API testing, database verification, and infrastructure validation."