7.5Lessons Learned

1. Flaky Test Reduction Was the Biggest Win

The Flaky Detector agent caught tests that humans had been ignoring for months. Running each test 5x and flagging inconsistencies eliminated accumulated tech debt.

How the Flaky Detector works:

class FlakyDetector:
    def detect(self, test_name: str, runs: int = 5) -> FlakyReport:
        results = []
        for i in range(runs):
            result = run_single_test(test_name)
            results.append(result)

        pass_count = sum(1 for r in results if r.passed)
        fail_count = runs - pass_count

        if 0 < fail_count < runs:  # Mix of pass and fail = flaky
            return FlakyReport(
                test_name=test_name,
                status="flaky",
                pass_rate=pass_count / runs,
                failure_reasons=self.analyze_failures(results)
            )
        return FlakyReport(test_name=test_name, status="stable")

Common flaky causes found:

• Tests depending on real network timeouts
• Tests sharing mutable state through global variables
• Tests depending on HashMap iteration order (Rust HashMaps are not ordered)
• Tests depending on file system timestamps

2. Agent Specialization Matters

Early attempts with a single "do everything" agent produced mediocre results. The single agent tried to analyze code, generate tests, run them, and fix failures all in one loop. It lost context, made inconsistent decisions, and produced lower-quality output than the specialized team.

Single agent (early approach): Average test quality score: 62/100 Eight specialized agents (final approach): Average test quality score: 84/100

3. The Test Fixer Agent Was the Most Complex

It needed to understand:

• Rust compiler errors (type mismatches, borrow checker violations)
• Test framework output (cargo test stdout/stderr format)
• The difference between "test bug" and "application bug"

The critical prompt for the Test Fixer:

You are fixing a failing Rust test. Determine whether this is:

A) A TEST BUG: the test code is wrong (wrong assertion, missing import,
   type mismatch, incorrect fixture setup). FIX the test.

B) AN APPLICATION BUG: the production code is wrong and the test correctly
   detected it. DO NOT fix the test. REPORT the application bug.

Error output:
{cargo_test_stderr}

Test code:
{test_code}

Production code:
{source_code}

4. Cost Control Required Explicit Budget Management

Without limits, agents would iterate endlessly on edge cases. They implemented a per-module token budget:

MODULE_BUDGETS = {
    "src/handlers/": 100_000,    # High-complexity, more budget
    "src/models/": 50_000,       # Medium complexity
    "src/utils/": 25_000,        # Low complexity, simple functions
}

When a module's budget was exhausted, agents stopped working on it and moved to the next module. This forced prioritization: high-complexity modules got more attention.