11.2The Spectrum

FULLY DETERMINISTIC <---------------------------------------> FULLY AUTONOMOUS
  (pytest scripts)                                              (free-roaming agent)

  |-- Recorded tests    |-- Parameterized       |-- Constrained        |-- Exploratory
  |   (fixed steps,     |   agents              |   agents             |   agents
  |    fixed data)      |   (fixed objective,   |   (fixed objective,  |   (no objective,
  |                     |    agent picks path)  |    bounded steps)    |    find anything)
  |                     |                       |                      |
  |   REGRESSION        |   TARGETED            |   SMOKE/SANITY       |   DISCOVERY
  |   Best for CI       |   Best for staging    |   Best for deploy    |   Best for sprint
  |   gates             |   environments        |   verification       |   exploration

Level 1: Fully Deterministic (Recorded Tests)

# Traditional test: every step is predetermined
def test_login():
    driver.navigate("https://app.example.com/login")
    driver.type("#email", "test@test.com")
    driver.type("#password", "password123")
    driver.click("#submit")
    assert driver.url == "https://app.example.com/dashboard"

Properties: Same input, same path, same result every time. Use for: CI gates, regression testing, deployability verification. Limitation: Breaks when the UI changes. No adaptability.

Level 2: Parameterized Agents

# Agent has a fixed objective but chooses its own path
def test_login_agent():
    agent = TestAgent(objective="Log in with test@test.com / password123")
    result = agent.run(max_steps=15)
    assert result.final_url == "https://app.example.com/dashboard"

Properties: Same objective, potentially different path, same expected outcome. Use for: Staging tests, self-healing regression tests. Limitation: May take different paths that have different performance characteristics.

Level 3: Constrained Agents

# Agent has a bounded scope but explores within it
def test_checkout_smoke():
    agent = TestAgent(
        objective="Complete a checkout with any valid product",
        config=HarnessConfig(max_steps=20, timeout_seconds=120)
    )
    result = agent.run()
    assert result.status == "pass"
    assert "order confirmation" in result.final_observation.lower()

Properties: Bounded exploration with a goal. Different paths, different products. Use for: Deploy verification, smoke tests, sanity checks. Limitation: Non-deterministic results make debugging harder.

Level 4: Fully Autonomous (Exploratory Agents)

# Agent explores freely, looking for anything interesting
def test_explore_app():
    agent = ExploratoryAgent(
        starting_url="https://app.example.com",
        config=HarnessConfig(max_steps=100, timeout_seconds=600)
    )
    findings = agent.explore()
    assert len(findings.critical_issues) == 0

Properties: No predetermined path. Agent discovers issues autonomously. Use for: Sprint exploration, security audits, discovering untested states. Limitation: Completely non-deterministic. Cannot be a CI gate.