Modern QA2026Key Design Decisions — tiles
Log inJoin
56 / 108 · 03 Agentic Testing Architectures · Case Study: The OpenObserve Council of Sub-Agents← prev⊞ allnext →☰ Read as one page

7.4Key Design Decisions

1. Shared Memory with Bounded Context

Each agent writes to a shared JSON state file, but only reads the sections relevant to its role. This prevents context pollution -- the Test Runner does not need to see the Code Analyzer's full AST output.

{
  "code_analysis": {
    "module": "src/handlers/search.rs",
    "functions": [
      {
        "name": "execute_search",
        "params": ["query: SearchQuery", "org_id: &str"],
        "return_type": "Result<SearchResponse, Error>",
        "error_paths": ["InvalidQuery", "PermissionDenied", "Timeout"],
        "complexity": "high"
      }
    ]
  },
  "test_generation": {
    "generated_tests": ["..."],
    "pending_review": ["..."]
  },
  "flaky_detection": {
    "flagged_tests": ["test_search_timeout -- passed 3/5 runs"]
  }
}

Why bounded context matters: When Agent 2 (Test Generator) reads the state, it only loads code_analysis and test_generation. It does not load flaky_detection or coverage_analysis. This keeps each agent's prompt focused and within token limits.

2. Human Approval Gates

Tests were not merged automatically. The PR Reviewer agent created a pull request with a structured summary, and a human made the final merge decision. This kept humans in the loop for quality control.

The PR summary format:

## Agent-Generated Test PR

**Module:** src/handlers/search.rs
**Tests Added:** 12
**Tests Modified:** 3
**Coverage Change:** 34% → 41% (+7 pp)

### New Tests
- test_execute_search_valid_query (happy path)
- test_execute_search_invalid_query (error: InvalidQuery)
- test_execute_search_permission_denied (error: PermissionDenied)
- ... (9 more)

### Flaky Tests Fixed
- test_search_timeout: added explicit timeout mock (was relying on real network)
- test_concurrent_search: added mutex for shared test state

### Reviewer Notes
- All tests pass 5/5 runs
- Naming convention matches existing tests
- Fixtures reuse existing test helpers

3. Incremental Execution

Agents did not regenerate the entire test suite each run. They analyzed what changed (new commits), identified what needed new tests, and generated incrementally.

class IncrementalAnalyzer:
    def identify_changes(self, since_commit: str) -> list[Change]:
        """Find what changed since the last agent run."""
        diff = git_diff(since_commit, "HEAD")
        changed_functions = []
        for file_change in diff:
            if file_change.path.endswith(".rs") and "test" not in file_change.path:
                # Source file changed -- needs test update
                changed_functions.extend(
                    self.extract_changed_functions(file_change)
                )
        return changed_functions

This kept token costs manageable. Instead of analyzing the entire codebase (millions of tokens), each run analyzed only the delta (thousands of tokens).