10.3ADR-003: Test Definition Format

Context

Tests need to be defined in a format the agent can read and execute. Options:

1. Natural language descriptions
2. Gherkin (Given/When/Then)
3. Structured YAML/JSON
4. Hybrid (structured with natural language steps)

Decision

Use structured YAML with natural language steps:

test: Login with valid credentials
preconditions:
  - User "test@example.com" exists with password "secret123"
steps:
  - Navigate to the login page
  - Enter email "test@example.com"
  - Enter password "secret123"
  - Click the login button
  - Verify the dashboard loads with welcome message
expected:
  - Dashboard page is displayed
  - Welcome message contains "test@example.com"
hints:  # Optional — the agent resolves concrete refs from snapshots at runtime
  login_page: https://app.example.com/login
  email_field: 'textbox "Email address"'
  password_field: 'textbox "Password"'
  submit_button: 'button "Sign in"'

Rationale

• Natural language steps give the agent freedom to adapt
• Structured format enables programmatic test management
• Hints use accessible role + name (the vocabulary of snapshot YAML), not CSS selectors — the agent maps them to element refs (e3, e5) from a live snapshot, so hints survive markup refactors
• The agent can ignore hints and discover the page from a fresh snapshot if they're stale

This format also plays well with Playwright's first-party workflow: it is deliberately close to the Markdown plans the planner agent writes into specs/ (see the Test Agents chapter), so migrating a suite in either direction is mechanical.