Architect-Level QA Interview: 20 Questions and Answers

Updated Aug 2026

Category 1: Architecture & Design

Q1: "Why would you choose agent-driven browser automation over traditional Playwright/Selenium?"

Answer: "Traditional automation is deterministic — you write exact steps, and they execute identically every time. That's great for regression but terrible for adaptability. When the UI changes, every affected test breaks.

Agent-driven automation adds a reasoning layer. The agent understands intent ('verify login works'), not just steps ('click button#submit'). When a selector changes, the agent can rediscover the element from a fresh page snapshot. When an unexpected dialog appears, the agent can dismiss it and continue.

The trade-off is non-determinism and cost. We mitigate non-determinism by logging every command and archiving Playwright traces for reproducibility, and we manage cost by driving the browser through the CLI-skill path rather than MCP — Microsoft's own benchmark puts that at roughly 4x fewer tokens per task."

Q2: "Walk me through the architecture of your test automation framework."

Answer: "Three layers:

Layer 1 — Test Definitions (Markdown/YAML): Natural language scenarios with optional hints, version-controlled and human-readable. With Playwright's test agents these live in specs/ — plans a product owner can read.

Layer 2 — Agent Orchestrator (a coding agent + skills): The agent reads test definitions, loads the browser-automation skill (a SKILL.md generated by playwright-cli install --skills), and executes through the shell. The agent decides execution strategy, handles failures, and reports results.

Layer 3 — Browser Infrastructure (Playwright engine via playwright-cli): The CLI drives the same battle-tested Playwright engine — auto-waiting, actionability checks, cross-browser. Page snapshots and screenshots land on disk in .playwright-cli/, not in the model's context. Named sessions give us parallel isolated browser contexts.

The key insight is that the agent operates at Layer 2, making intelligent decisions, while Layer 3 handles the mechanical complexity of browser control. The skill is the bridge — a ~100-line markdown file that gives the agent all the domain knowledge it needs."

Q3: "How does the browser-automation skill actually work under the hood?"

Answer: "When the agent decides browser automation is needed, it invokes the skill. The system loads a markdown file — SKILL.md, generated by playwright-cli install --skills — into the agent's conversation context. This file documents the CLI's command surface.

The agent then executes commands via the shell: playwright-cli open <url>, playwright-cli snapshot, playwright-cli click e14. The snapshot command writes a compact YAML accessibility summary to disk with element refs like e14; the agent reads that file only when it needs it, and interacts by ref instead of authoring CSS selectors.

Underneath, it's the standard Playwright engine performing the action with full auto-waiting and actionability checks. The whole chain: Skill loads markdown → agent sends shell command → CLI drives the Playwright engine → engine waits for actionability → browser executes → a minimal response (often just a file path) comes back."

Q4: "How do you handle test flakiness in an AI-driven framework?"

Answer: "We distinguish between three types of 'flakiness':

True flakiness (timing issues): Playwright's actionability checks handle this — every click/fill auto-waits for the element to be visible, stable, not obscured, and enabled. This eliminates 80% of traditional flakiness.

Infrastructure flakiness (network, browser crashes): Fresh browser sessions per test group in CI. For network issues, we set explicit timeouts and capture failure artifacts — screenshot, page snapshot, and the full Playwright trace — so the agent can reason about what happened.

Test logic flakiness (non-deterministic agent behavior): We log every command the agent executes. If a test passes inconsistently, we review the command log and trace to see where the agent's reasoning diverged. We then add hints or more explicit steps to constrain the agent's decisions.

The self-healing aspect actually reduces flakiness compared to traditional frameworks — when a selector breaks, the agent rediscovers the element from a fresh snapshot instead of failing."

Q5: "What's your CI/CD integration strategy?"

Answer: "Headless browsers, isolated sessions, GitHub Actions matrix strategy.

Each test group (auth, dashboard, checkout) runs as a separate matrix job with its own browser session. On failure, we capture screenshots, snapshots, and Playwright traces (which include network capture) as artifacts with 30-day retention — the trace is the audit trail of exactly what the agent did.

We output JUnit XML for dashboard integration and a JSON report for programmatic analysis. Playwright's healer agent runs as a follow-up step on failures: it replays the failing step, inspects the UI, and either proposes a patch or concludes the app itself is broken — that verdict gates whether we auto-retry or page a human.

For parallelization, named sessions (playwright-cli -s=<name>) give each worker an isolated browser context within a runner."

Category 2: Technical Deep Dives

Q6: "Explain actionability checks. Where should they be implemented?"

Answer: "Five checks run before every interaction — the model Playwright pioneered:

1. Visible — Non-zero dimensions, not display:none or visibility:hidden
2. Stable — Position unchanged across animation frames (catches CSS animations)
3. ReceivesEvents — hit-testing at the element's center reaches the target, not a covering overlay
4. Enabled — Not disabled, aria-disabled, or in a disabled fieldset
5. Editable — (for text input) Accepts input, not readonly

They run in a polling loop until all pass or a timeout fires, and a good implementation tells you which check failed — 'obscured by div.modal-overlay' makes debugging trivial.

The architectural question interviewers like: where should these live? Playwright implements them in each client library. Vibium — Jason Huggins' AI-native tool — made the interesting choice of implementing them once, server-side in a Go binary, so every client gets identical behavior for free. Knowing both designs and the trade-off (per-client flexibility vs single-implementation consistency) is the architect-level answer."

Q7: "What is WebDriver BiDi and why does it matter?"

Answer: "WebDriver BiDi is a W3C standard that combines the best of two predecessors. Classic WebDriver was standardized and cross-browser but one-directional — HTTP request/response, no events. CDP was bidirectional with rich events but Chrome-specific and unstable.

BiDi uses WebSocket for bidirectional JSON messaging — both commands from client to browser AND events pushed from browser to client. It's governed by the W3C with buy-in from all major browser vendors.

And as of 2026 it's the present, not the future: it covers roughly 70% of the CDP surface across Chrome and Firefox, WebdriverIO v9 defaults to it, and Selenium 4 exposes it at the low level with high-level APIs slated for Selenium 5. For our stack it means standards-based, future-proof, cross-browser automation with real-time events — console logs and network activity pushed to us as they happen."

Q8: "How does the CLI approach keep token usage low? Where's the catch?"

Answer: "The mechanism is disk-first state. With MCP, every action streams the page's accessibility tree — often thousands of tokens — into the model's context whether the agent needs it or not. With the CLI, snapshots and screenshots are written to .playwright-cli/ and the agent gets back a file path. The agent reads state on demand.

Microsoft's benchmark: ~27,000 tokens per typical task via CLI versus ~114,000 via MCP — about 4x, with longer sessions reporting up to 10x.

The catch is latency: reading state from disk adds round-trips, so a CLI session can be slower in wall-clock time even while far cheaper in tokens. And in sandboxed environments with no filesystem, MCP is the only option. Token efficiency and latency are different axes — you pick per context, which is exactly what Microsoft's own guidance says."

Q9: "Explain the token economics of skills vs MCP for browser automation."

Answer: "The naive arithmetic makes skills look absurdly better: a SKILL.md is ~1,000 tokens once, a shell command ~30 tokens, versus thousands per MCP action from tool schemas plus accessibility trees. Per-step, that's a double-digit multiple.

The honest, measured number is smaller and more defensible: Microsoft benchmarks a typical end-to-end task at ~27k tokens via CLI versus ~114k via MCP — roughly 4x. The gap between naive and measured matters: in a real task the agent still reads snapshots from disk when it needs them. The saving comes from reading state on demand instead of receiving it on every action, not from never seeing state.

4x is still decisive at scale: it's the difference between finishing a long test session with room to reason and compacting context halfway through. And it compounds with cost — across hundreds of CI runs a day, the CLI path is what keeps agent-driven testing economically viable."

Q10: "How do you handle self-healing tests?"

Answer: "Three tiers:

Tier 1 (most failures, cheapest): When an interaction fails, the agent takes a fresh playwright-cli snapshot and rediscovers the element by role and accessible name in the YAML — element refs come from snapshots, so a re-snapshot is the recovery mechanism.

Tier 2: The agent takes a screenshot and reads the page state to understand what happened — loading spinners, redirects, unexpected dialogs — then acts accordingly.

Tier 3 (productized): Playwright's healer agent replays the failing step against the live app, patches locators, waits, or data, and reruns until green — or concludes the functionality itself is broken.

That last capability is the standard I hold any self-healing system to: a healer that always makes the test pass is a bug-hiding machine. We track healing events in a log. High healing rates on one test mean its assumptions need updating. High healing rates across tests mean a major UI refactor happened. Self-healing should fix stale selectors, not mask real regressions."

Category 3: Strategy & Trade-offs

Q11: "When would you NOT use agent-driven testing?"

Answer: "Three scenarios:

1. Performance testing — You need deterministic, repeatable measurements. Agent reasoning overhead per step is unacceptable for load testing.

2. Trivial regression checks — 'Does the homepage return 200?' doesn't need AI. A simple curl check is faster, cheaper, and more reliable.

3. Compliance testing with audit requirements — Some regulations require test scripts to be deterministic and reproducible. Agent reasoning introduces variability that auditors may not accept — though archived Playwright traces (every action, every snapshot, network included) go a long way toward satisfying 'show me what the AI did.'

For these, traditional Playwright/Selenium scripts are better. The sweet spot for agents is complex functional flows, exploratory testing, and tests that need to adapt to changing UIs."

Q12: "How would you convince a skeptical architect that AI testing is production-ready?"

Answer: "I'd address the three common objections:

'AI is non-deterministic.' True, but we mitigate it by logging every command and archiving traces for reproducibility. In practice, the same test produces the same sequence of commands the vast majority of the time because the agent follows SKILL.md instructions — it's only on failure recovery that reasoning diverges. And the strongest argument: Microsoft now ships first-party test agents (planner/generator/healer) in Playwright itself. This is no longer a fringe pattern.

'It's too expensive.' With the CLI-skill path, browser control costs ~4x fewer tokens than MCP by Microsoft's own benchmark. Compare total token spend to the engineering hours saved on test maintenance — even one day of a QA engineer's time pays for a lot of agent runs.

'It's too slow.' Browser commands execute at engine speed; the agent's reasoning adds overhead per decision, not per wait. Traditional suites are full of padded sleeps; agent-driven tests wait exactly as long as actionability requires. Wall-clock is roughly comparable, and the maintenance loop is dramatically faster.

Then I'd show the real data from our own suite: maintenance time trends, healing rates, cost per run."

Q13: "How do you handle test data management?"

Answer: "Three approaches depending on isolation needs:

Inline test data: For simple tests, data is embedded in the test definition. playwright-cli fill e8 "test@example.com" — the value is right there in the plan.

API-driven setup: For tests that need specific state (user accounts, order history), we call the app's API before browser tests to create the required data — via curl or playwright-cli eval 'fetch(...)'.

Database seeding: For CI, we run migration scripts that create a known state before the test suite. Each test group gets its own database or schema partition for isolation.

The agent doesn't manage test data directly — it focuses on UI interaction. Data setup is handled by scripts in the framework's /scripts/ directory."

Q14: "What does your testing pyramid look like with AI?"

Answer:

        /\
       /  \   Agent-driven E2E (browser tests via CLI skill + test agents)
      /    \  ~50 tests, critical user journeys
     /______\
    /        \  API/Integration tests (traditional)
   /          \ ~200 tests, business logic verification
  /____________\
 /              \ Unit tests (traditional)
/________________\ ~2000+ tests, code correctness

"AI-driven testing sits at the top of the pyramid — the smallest number of tests with the highest coverage per test. We don't use AI for unit tests (deterministic, no browser needed) or most API tests (no UI involved). The agent adds value where human judgment is needed: complex flows, visual verification, adaptive interaction. The planner agent also feeds the middle layers: scenarios it discovers often become cheaper API-level tests."

Category 4: Practical Scenarios

Q15: "Walk me through debugging a failing browser test."

Answer: "The failure artifacts give us everything:

1. Command log — I see the exact sequence of CLI commands. Step 7 was playwright-cli click e17 which timed out.

2. Trace — I open the Playwright trace: every action with before/after snapshots, console output, and network capture. I can see a modal overlay appeared right before the click.

3. Snapshot + screenshot — The saved YAML snapshot shows a dialog node ('Are you sure you want to proceed?') that the test didn't expect; the screenshot confirms it visually.

4. Fix — I add a step to handle the confirmation dialog before the original click, or update the spec to expect it.

If the healer agent ran, it might have already patched this — but we still review its diff to decide whether the plan should change. Healer patches are code review items, not silent fixes."

Q16: "How would you test a Single Page Application?"

Answer: "SPAs have specific challenges: navigation doesn't trigger full page loads, content renders asynchronously, and URLs may not change.

Key techniques:

1. Trust actionability, not page load events. The engine's auto-wait means a click on a ref proceeds only when the element is actually interactable — that absorbs most SPA timing issues without explicit waits.

2. Snapshot after state changes:

playwright-cli click e12          # client-side route change
playwright-cli snapshot           # fresh snapshot — new refs for the new view

Element refs are per-snapshot; after a route transition you re-snapshot rather than reuse stale refs — which is exactly the discipline SPAs require anyway.

3. Verify content, not URL. Client-side routing may not change the URL meaningfully; assert on what the snapshot shows (headings, roles, text), not the address bar.

4. Handle lazy loading by scrolling and re-snapshotting to confirm the lazily rendered components exist."

Q17: "How do you handle authentication across tests?"

Answer: "Three strategies:

Strategy 1 — Login via UI (realistic, slow): Drive the actual login form once: open, snapshot, fill email/password refs, press Enter, verify the dashboard appears.

Strategy 2 — Saved state (fast, reliable):

# Once, after a real login:
playwright-cli state-save auth-state

# Every other test:
playwright-cli state-load auth-state
playwright-cli goto https://app.example.com/dashboard   # already authenticated

Strategy 3 — API auth (for CI): Fetch a token via the API, inject it with playwright-cli eval into localStorage or cookies, then navigate.

We use Strategy 1 for the actual login test and Strategy 2 for everything else — one thorough login verification plus fast, deterministic setup for the rest. This is also the seed-test pattern Playwright's test agents formalize: seed.spec.ts provides the authenticated context every agent inherits."

Q18: "How do you handle tests that interact with third-party services?"

Answer: "Three approaches:

1. Mock at the network level: intercept fetch() via playwright-cli eval (or route interception in the underlying framework) so calls to the third party return canned responses.

2. Use test/sandbox environments: Most payment providers (Stripe, PayPal) have sandbox modes. Configure the app to use sandbox credentials in the test environment.

3. Stop at the boundary: Test up to the point of third-party interaction, verify the outgoing request payload (the trace's network capture shows it), then skip the actual external call."

Q19: "What metrics do you track for your test automation?"

Answer: "Five key metrics:

1. Test reliability rate — % of tests that pass consistently (target: >98%)
2. Self-healing rate — % of runs where the agent recovered from a failure (track over time — should decrease as the suite stabilizes; every healer patch is reviewed)
3. Execution time — Per-test and per-suite (detect performance regressions)
4. Token cost — Per-test and per-suite (budget management; the reason we're on the CLI path)
5. Failure-to-fix time — How long between a test failure and the fix being merged (measures the value of failure artifacts)

We track these in a JSON report per run and graph trends weekly. A spike in healing rate means a UI deployment changed something. A spike in execution time means either the app got slower or a test got stuck."

Q20: "Where do you see AI-driven testing going in the next 2-3 years?"

Answer: "Three directions:

1. From assistants to first-party agents — already happened. Playwright ships planner/generator/healer agents today; the debate moved from 'should AI write tests?' to 'how do we review what it writes?' Expect Cypress (cy.prompt()) and the commercial platforms to keep converging on the same plan → generate → heal loop.

2. Persistent application memory. Today's agents rediscover the app every session. The next frontier is an 'app map' the agent maintains across sessions — Vibium's planned Cortex (SQLite + embeddings) is the clearest articulation, but whoever ships it changes the economics of exploratory agent testing. Watch this space.

3. Autonomous QA at the suite level. Agents that don't just execute predefined tests but explore, identify risks, and grow the suite — OpenObserve's 'Council of Sub Agents' (8 specialized agents growing a suite from 380 to 700+ tests) is an early production example.

The agent skills pattern — lightweight markdown teaching agents domain knowledge — has become the standard interface, portable across Claude Code, Cursor, Gemini CLI, and Codex CLI. Browser automation is just one domain; the same pattern applies to API testing, database verification, and infrastructure validation."