17.1Category 1: Architecture & Design

Q1: "Why would you choose agent-driven browser automation over traditional Playwright/Selenium?"

Answer: "Traditional automation is deterministic — you write exact steps, and they execute identically every time. That's great for regression but terrible for adaptability. When the UI changes, every affected test breaks.

Agent-driven automation adds a reasoning layer. The agent understands intent ('verify login works'), not just steps ('click button#submit'). When a selector changes, the agent can rediscover the element from a fresh page snapshot. When an unexpected dialog appears, the agent can dismiss it and continue.

The trade-off is non-determinism and cost. We mitigate non-determinism by logging every command and archiving Playwright traces for reproducibility, and we manage cost by driving the browser through the CLI-skill path rather than MCP — Microsoft's own benchmark puts that at roughly 4x fewer tokens per task."

Q2: "Walk me through the architecture of your test automation framework."

Answer: "Three layers:

Layer 1 — Test Definitions (Markdown/YAML): Natural language scenarios with optional hints, version-controlled and human-readable. With Playwright's test agents these live in specs/ — plans a product owner can read.

Layer 2 — Agent Orchestrator (a coding agent + skills): The agent reads test definitions, loads the browser-automation skill (a SKILL.md generated by playwright-cli install --skills), and executes through the shell. The agent decides execution strategy, handles failures, and reports results.

Layer 3 — Browser Infrastructure (Playwright engine via playwright-cli): The CLI drives the same battle-tested Playwright engine — auto-waiting, actionability checks, cross-browser. Page snapshots and screenshots land on disk in .playwright-cli/, not in the model's context. Named sessions give us parallel isolated browser contexts.

The key insight is that the agent operates at Layer 2, making intelligent decisions, while Layer 3 handles the mechanical complexity of browser control. The skill is the bridge — a ~100-line markdown file that gives the agent all the domain knowledge it needs."

Q3: "How does the browser-automation skill actually work under the hood?"

Answer: "When the agent decides browser automation is needed, it invokes the skill. The system loads a markdown file — SKILL.md, generated by playwright-cli install --skills — into the agent's conversation context. This file documents the CLI's command surface.

The agent then executes commands via the shell: playwright-cli open <url>, playwright-cli snapshot, playwright-cli click e14. The snapshot command writes a compact YAML accessibility summary to disk with element refs like e14; the agent reads that file only when it needs it, and interacts by ref instead of authoring CSS selectors.

Underneath, it's the standard Playwright engine performing the action with full auto-waiting and actionability checks. The whole chain: Skill loads markdown → agent sends shell command → CLI drives the Playwright engine → engine waits for actionability → browser executes → a minimal response (often just a file path) comes back."

Q4: "How do you handle test flakiness in an AI-driven framework?"

Answer: "We distinguish between three types of 'flakiness':

True flakiness (timing issues): Playwright's actionability checks handle this — every click/fill auto-waits for the element to be visible, stable, not obscured, and enabled. This eliminates 80% of traditional flakiness.

Infrastructure flakiness (network, browser crashes): Fresh browser sessions per test group in CI. For network issues, we set explicit timeouts and capture failure artifacts — screenshot, page snapshot, and the full Playwright trace — so the agent can reason about what happened.

Test logic flakiness (non-deterministic agent behavior): We log every command the agent executes. If a test passes inconsistently, we review the command log and trace to see where the agent's reasoning diverged. We then add hints or more explicit steps to constrain the agent's decisions.

The self-healing aspect actually reduces flakiness compared to traditional frameworks — when a selector breaks, the agent rediscovers the element from a fresh snapshot instead of failing."

Q5: "What's your CI/CD integration strategy?"

Answer: "Headless browsers, isolated sessions, GitHub Actions matrix strategy.

Each test group (auth, dashboard, checkout) runs as a separate matrix job with its own browser session. On failure, we capture screenshots, snapshots, and Playwright traces (which include network capture) as artifacts with 30-day retention — the trace is the audit trail of exactly what the agent did.

We output JUnit XML for dashboard integration and a JSON report for programmatic analysis. Playwright's healer agent runs as a follow-up step on failures: it replays the failing step, inspects the UI, and either proposes a patch or concludes the app itself is broken — that verdict gates whether we auto-retry or page a human.

For parallelization, named sessions (playwright-cli -s=<name>) give each worker an isolated browser context within a runner."