CI/CD Integration for AI-Driven Browser Tests

Updated Aug 2026

Principles

1. Headless in CI — no display available; --headed is a local-development flag
2. Fresh session per test — one named session (-s=name) per test for isolation, closed at the end
3. Artifacts on failure — screenshots, YAML snapshots, agent logs, and Playwright traces where produced
4. Deterministic timeouts — no infinite waits
5. Exit codes matter — CI gates on pass/fail

GitHub Actions Configuration

Basic Setup

name: Browser Tests
on: [push, pull_request]

jobs:
  browser-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '24'

      - name: Install Playwright CLI
        run: |
          npm install -g @playwright/cli@latest
          playwright-cli install          # initialize the .playwright-cli/ workspace

      - name: Install browsers + system dependencies
        run: npx playwright install --with-deps chromium

      - name: Smoke test
        run: |
          playwright-cli open https://staging.example.com
          playwright-cli eval "document.querySelector('h1').textContent" | grep -q "Welcome"
          playwright-cli close

      - name: Upload failure artifacts
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: test-failures
          path: failures/

Two things changed from the 2025-era setups worth noticing: browser and system dependencies come from Playwright's own installer (npx playwright install --with-deps) instead of a hand-maintained apt-get list of Chrome libraries, and there are no tool-specific environment variables — session lifecycle is explicit in the commands.

Full Test Suite

name: Full Test Suite
on:
  push:
    branches: [main]
  pull_request:

jobs:
  browser-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        test-group: [auth, dashboard, checkout, settings]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'

      - name: Install dependencies
        run: |
          npm install -g @playwright/cli@latest
          playwright-cli install
          npx playwright install --with-deps chromium

      - name: Run ${{ matrix.test-group }} tests
        env:
          TEST_BASE_URL: ${{ secrets.STAGING_URL }}
        run: ./run-tests.sh ${{ matrix.test-group }}

      - name: Upload screenshots
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: screenshots-${{ matrix.test-group }}
          path: screenshots/

      - name: Upload failure details
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: failures-${{ matrix.test-group }}
          path: failures/

Test Runner Script

run-tests.sh

#!/bin/bash
set -euo pipefail

TEST_GROUP="${1:-all}"
BASE_URL="${TEST_BASE_URL:-http://localhost:3000}"
FAILURES_DIR="failures"
SCREENSHOTS_DIR="screenshots"

mkdir -p "$FAILURES_DIR" "$SCREENSHOTS_DIR"

PASSED=0
FAILED=0
TOTAL=0

# pw: run a CLI command inside this test's named session
pw() { playwright-cli -s="$CURRENT_TEST" "$@"; }
export -f pw
export BASE_URL

run_test() {
  local test_name="$1"
  local test_script="$2"
  TOTAL=$((TOTAL + 1))
  export CURRENT_TEST="$test_name"

  echo -n "  $test_name ... "

  # Execute test in its own session, capture output
  if output=$(bash -c "$test_script" 2>&1); then
    echo "PASS"
    PASSED=$((PASSED + 1))
  else
    echo "FAIL"
    FAILED=$((FAILED + 1))

    # Capture failure artifacts
    mkdir -p "$FAILURES_DIR/$test_name"
    echo "$output" > "$FAILURES_DIR/$test_name/output.txt"
    pw screenshot 2>/dev/null || true              # PNG lands in .playwright-cli/
    pw snapshot 2>/dev/null || true                # YAML a11y state
    pw eval "location.href" > "$FAILURES_DIR/$test_name/current_url.txt" 2>/dev/null || true
    cp -r .playwright-cli/ "$FAILURES_DIR/$test_name/workspace/" 2>/dev/null || true
  fi

  # Fresh session per test: always tear down
  pw close 2>/dev/null || true
}

echo "Running $TEST_GROUP tests against $BASE_URL"
echo "=========================================="

# Load and run tests for the group
case $TEST_GROUP in
  auth)
    run_test "login_valid" '
      pw open "$BASE_URL/login" &&
      pw snapshot &&
      pw fill e3 "test@example.com" &&
      pw fill e4 "password123" &&
      pw click e5 &&
      pw eval "document.querySelector(\"h1\").textContent" | grep -q "Dashboard"
    '

    run_test "login_invalid" '
      pw open "$BASE_URL/login" &&
      pw snapshot &&
      pw fill e3 "wrong@example.com" &&
      pw fill e4 "wrongpass" &&
      pw click e5 &&
      pw eval "document.querySelector(\".error\").textContent" | grep -qi "invalid"
    '
    ;;

  dashboard)
    run_test "dashboard_loads" '
      pw open "$BASE_URL/dashboard" &&
      pw eval "document.querySelector(\".metric-count\").textContent" | grep -qE "[0-9]+"
    '
    ;;

  *)
    echo "Unknown test group: $TEST_GROUP"
    exit 1
    ;;
esac

echo ""
echo "=========================================="
echo "Results: $PASSED passed, $FAILED failed, $TOTAL total"

# Exit with failure if any tests failed
[ "$FAILED" -eq 0 ]

A candid note on this script: hard-coding refs (e3, e4) in a shell script works only for pages whose snapshot layout is stable. That's fine for smoke tests. The moment refs drift, you want either the agent in the loop (it re-snapshots and re-resolves) or promoted, generated .spec.ts tests from the generator agent — scripted ref sequences are the least self-healing artifact in this stack. Use them deliberately.

Docker Configuration

Dockerfile for Test Runner

FROM node:24-slim

# Install Playwright CLI + browsers with system dependencies
RUN npm install -g @playwright/cli@latest \
    && npx playwright install --with-deps chromium

# Copy test scripts
WORKDIR /tests
COPY . .

# Initialize the workspace
RUN playwright-cli install

CMD ["./run-tests.sh", "all"]

(You can also start from Microsoft's official Playwright image — mcr.microsoft.com/playwright:v1.61.0-noble — which ships browsers and dependencies preinstalled.)

docker-compose.yml (with test app)

version: '3.8'
services:
  app:
    build: ./app
    ports:
      - "3000:3000"
    healthcheck:
      test: curl -f http://localhost:3000/health
      interval: 5s
      timeout: 3s
      retries: 5

  tests:
    build:
      context: ./tests
      dockerfile: Dockerfile
    depends_on:
      app:
        condition: service_healthy
    environment:
      - TEST_BASE_URL=http://app:3000
    volumes:
      - ./test-results:/tests/failures

Parallel Execution in CI

Matrix Strategy (GitHub Actions)

strategy:
  fail-fast: false
  matrix:
    test-group: [auth, dashboard, checkout, settings, admin]

Each test group runs as a separate job with its own browser instance. fail-fast: false ensures all groups run even if one fails.

Within a Single Job

# Run 4 tests in parallel — isolation comes from one named session per test
cat test_list.txt | xargs -P4 -I{} bash -c '
  ./run-single-test.sh "{}"     # opens/closes playwright-cli -s="{}"
'

Resource Limits

CI runner recommendation: 4 parallel workers on a standard 2-core runner (the browser is mostly I/O-bound, not CPU-bound).

Artifact Management

What to Capture

The trace row is the important 2026 addition. If an AI agent drove a browser in your pipeline and you cannot answer "show me exactly what it did," you have an unaccountable system. Traces are the literal answer to that question — archive them for every agent-driven run, and feed them back to the agent (or the healer) when a run needs debugging.

Retention Policy

# GitHub Actions
- uses: actions/upload-artifact@v4
  with:
    name: test-results
    path: results/
    retention-days: 30    # Keep for 30 days; consider longer for traces if compliance asks

Interview Talking Point

"Our CI pipeline runs browser tests through the Playwright CLI — headless, one named session per test for isolation, torn down after each run. Browsers and system deps come from npx playwright install --with-deps, so there's no hand-maintained library list. We parallelize with a GitHub Actions matrix across test groups, and within a job by giving each worker its own session name. On failure we capture a screenshot, the YAML accessibility snapshot, and the current URL; for agent-driven and healer runs we also archive Playwright traces, which since 1.60 include HAR network capture — that's our audit trail for what the AI actually did in the browser. The scripted smoke layer is deliberately thin: anything that needs resilience is either agent-executed or promoted to generated Playwright specs."