1.7Interview Depth Check

Question 1

Prompt: Your team has a 500-test Selenium suite with a 25% flaky rate. A VP asks you to "just switch to AI agents." How do you respond, and what do you actually do? What a strong answer should cover:

• Why wholesale replacement is wrong (cost, non-determinism in CI gates, debugging complexity)
• A triage approach: classify tests by stability and business criticality
• Where agents add value (self-healing for flaky selectors, exploratory coverage)
• A phased rollout plan with measurable success criteria Example answer:
• I would first push back on the "just switch" framing -- replacing a 500-test suite wholesale introduces massive risk. Instead, I would classify the flaky tests by root cause: broken selectors, timing issues, environment state, or genuine bugs.
• For the selector-breakage bucket, agentic tests with self-healing capability are a strong fit. For timing issues, better wait strategies in existing scripts may be cheaper than agents. For environment state, neither approach helps -- we need test data isolation.
• I would propose a pilot: convert the 10 highest-maintenance flaky tests to agents, measure maintenance hours saved versus token cost, and present the data before scaling further.

Question 2

Prompt: A colleague argues that since agents are non-deterministic, they can never be trusted in a CI pipeline. Do you agree or disagree, and what evidence supports your position? What a strong answer should cover:

• Acknowledgment that raw agents are indeed non-deterministic
• Techniques to tame non-determinism: outcome assertions, decision caching, temperature=0
• The distinction between path determinism and outcome determinism
• Real-world examples of agents operating reliably in CI with guardrails Example answer:
• I partially agree -- an unconstrained agent with no guardrails should not gate a deployment. But the claim that agents can never work in CI conflates path non-determinism with outcome non-determinism.
• If I assert on the final page state rather than the exact click sequence, the agent can take different paths and still produce a reliable pass/fail signal. Combined with temperature=0, max-step limits, and domain allowlists, agents become predictable enough for staging CI gates.
• I would not use agents for the final production deployment gate, but for pre-merge validation in staging, they work well with proper harness configuration.

Question 3

Prompt: You are designing a test strategy for a brand-new e-commerce application. Walk me through how you would decide which tests are scripts, which are agents, and which are manual. What a strong answer should cover:

• A structured decision framework (business criticality, UI change frequency, state space complexity)
• Specific examples mapped to each category
• Cost and speed considerations
• Where manual testing still wins (usability, subjective quality) Example answer:
• I start with the critical happy paths: login, search, add-to-cart, checkout, payment. These gate every deployment, so they must be deterministic scripts -- fast, cheap, debuggable.
• For areas with high UI churn (promotional landing pages, A/B-tested flows, new feature iterations), I use constrained agents. They adapt to layout changes without breaking.
• Exploratory agents run nightly against the full application, looking for broken links, console errors, and unexpected states. These produce findings reports, not pass/fail gates.
• Manual testing covers subjective quality: does the checkout flow feel trustworthy? Is the error messaging helpful? No automation handles that well yet.