17.3Category 3: Strategy & Trade-offs

Q11: "When would you NOT use agent-driven testing?"

Answer: "Three scenarios:

1. Performance testing — You need deterministic, repeatable measurements. Agent reasoning overhead per step is unacceptable for load testing.

2. Trivial regression checks — 'Does the homepage return 200?' doesn't need AI. A simple curl check is faster, cheaper, and more reliable.

3. Compliance testing with audit requirements — Some regulations require test scripts to be deterministic and reproducible. Agent reasoning introduces variability that auditors may not accept — though archived Playwright traces (every action, every snapshot, network included) go a long way toward satisfying 'show me what the AI did.'

For these, traditional Playwright/Selenium scripts are better. The sweet spot for agents is complex functional flows, exploratory testing, and tests that need to adapt to changing UIs."

Q12: "How would you convince a skeptical architect that AI testing is production-ready?"

Answer: "I'd address the three common objections:

'AI is non-deterministic.' True, but we mitigate it by logging every command and archiving traces for reproducibility. In practice, the same test produces the same sequence of commands the vast majority of the time because the agent follows SKILL.md instructions — it's only on failure recovery that reasoning diverges. And the strongest argument: Microsoft now ships first-party test agents (planner/generator/healer) in Playwright itself. This is no longer a fringe pattern.

'It's too expensive.' With the CLI-skill path, browser control costs ~4x fewer tokens than MCP by Microsoft's own benchmark. Compare total token spend to the engineering hours saved on test maintenance — even one day of a QA engineer's time pays for a lot of agent runs.

'It's too slow.' Browser commands execute at engine speed; the agent's reasoning adds overhead per decision, not per wait. Traditional suites are full of padded sleeps; agent-driven tests wait exactly as long as actionability requires. Wall-clock is roughly comparable, and the maintenance loop is dramatically faster.

Then I'd show the real data from our own suite: maintenance time trends, healing rates, cost per run."

Q13: "How do you handle test data management?"

Answer: "Three approaches depending on isolation needs:

Inline test data: For simple tests, data is embedded in the test definition. playwright-cli fill e8 "test@example.com" — the value is right there in the plan.

API-driven setup: For tests that need specific state (user accounts, order history), we call the app's API before browser tests to create the required data — via curl or playwright-cli eval 'fetch(...)'.

Database seeding: For CI, we run migration scripts that create a known state before the test suite. Each test group gets its own database or schema partition for isolation.

The agent doesn't manage test data directly — it focuses on UI interaction. Data setup is handled by scripts in the framework's /scripts/ directory."

Q14: "What does your testing pyramid look like with AI?"

Answer:

        /\
       /  \   Agent-driven E2E (browser tests via CLI skill + test agents)
      /    \  ~50 tests, critical user journeys
     /______\
    /        \  API/Integration tests (traditional)
   /          \ ~200 tests, business logic verification
  /____________\
 /              \ Unit tests (traditional)
/________________\ ~2000+ tests, code correctness

"AI-driven testing sits at the top of the pyramid — the smallest number of tests with the highest coverage per test. We don't use AI for unit tests (deterministic, no browser needed) or most API tests (no UI involved). The agent adds value where human judgment is needed: complex flows, visual verification, adaptive interaction. The planner agent also feeds the middle layers: scenarios it discovers often become cheaper API-level tests."