8.2The Five Essential Gates

Gate 1: All Tests Pass

The most basic gate. If any test fails, the suite is not ready.

# Run all tests with verbose output and short tracebacks
pytest tests/ -v --tb=short

This catches:

• Import errors from hallucinated modules
• Missing fixtures or helpers
• Assertion failures from incorrect expected values
• Runtime errors from nonexistent methods

Common issue with AI tests: The first run often has 10-20% failures due to import errors and missing fixtures. Fix these mechanically before proceeding to deeper review.

Gate 2: No Tests Are Empty or Assertion-Free

A test function with no assert statement is worthless -- it only proves the code does not crash, not that it behaves correctly.

# Automated check: find assertion-free test functions
import ast
import sys
from pathlib import Path

def find_assertionless_tests(test_dir: str) -> list[str]:
    """Find test functions that contain no assert statements."""
    violations = []

    for path in Path(test_dir).rglob("test_*.py"):
        tree = ast.parse(path.read_text())
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
                if not node.name.startswith("test_"):
                    continue

                has_assert = any(
                    isinstance(child, ast.Assert)
                    or (isinstance(child, ast.Expr)
                        and isinstance(child.value, ast.Call)
                        and _is_assertion_call(child.value))
                    for child in ast.walk(node)
                )

                if not has_assert:
                    violations.append(f"{path}:{node.lineno} - {node.name}")

    return violations

def _is_assertion_call(call_node: ast.Call) -> bool:
    """Check if a call is to pytest.raises, expect, or similar."""
    if isinstance(call_node.func, ast.Attribute):
        return call_node.func.attr in ("raises", "warns", "approx")
    return False

if __name__ == "__main__":
    violations = find_assertionless_tests("tests/")
    if violations:
        print("GATE FAILED: Tests without assertions:")
        for v in violations:
            print(f"  {v}")
        sys.exit(1)
    print("GATE PASSED: All tests contain assertions")

Gate 3: Coverage Did Not Decrease

AI-generated tests should increase or maintain coverage, never decrease it. This gate prevents a scenario where new tests are added but existing tests are accidentally deleted or broken.

# Run with coverage enforcement
pytest --cov=app --cov-fail-under=80 --cov-report=term-missing

# For stricter enforcement: compare against a baseline
pytest --cov=app --cov-report=json
python -c "
import json
current = json.load(open('coverage.json'))['totals']['percent_covered']
baseline = 82.5  # Store this in a config file or environment variable
if current < baseline:
    print(f'GATE FAILED: Coverage dropped from {baseline}% to {current}%')
    exit(1)
print(f'GATE PASSED: Coverage at {current}% (baseline: {baseline}%)')
"

Gate 4: No New Flaky Tests

Flaky tests are tests that sometimes pass and sometimes fail without any code change. AI-generated tests are particularly prone to flakiness because of non-deterministic dependencies (time, random data, external services).

# Run the test suite 3 times and fail if any test is inconsistent
pytest tests/ --count=3 -x

# For a more thorough check, use pytest-repeat
pytest tests/ --count=5 --repeat-scope=session -x

# Or use a dedicated flaky test detector
pytest tests/ -p no:randomly --count=3 2>&1 | python detect_flaky.py

detect_flaky.py:

import sys
import re
from collections import defaultdict

results = defaultdict(list)
current_run = 0

for line in sys.stdin:
    if "PASSED" in line or "FAILED" in line:
        test_name = re.search(r'(test_\w+)', line)
        if test_name:
            status = "PASS" if "PASSED" in line else "FAIL"
            results[test_name.group(1)].append(status)

flaky = {
    name: statuses
    for name, statuses in results.items()
    if len(set(statuses)) > 1  # Mix of PASS and FAIL
}

if flaky:
    print("GATE FAILED: Flaky tests detected:")
    for name, statuses in flaky.items():
        print(f"  {name}: {statuses}")
    sys.exit(1)

print("GATE PASSED: No flaky tests detected")

Gate 5: Mutation Score Check (Optional but Powerful)

Mutation testing modifies your source code (e.g., changing > to >=, + to -) and checks whether your tests catch the change. A high mutation score means your tests actually detect bugs.

# Run mutation testing with mutmut (Python)
mutmut run --paths-to-mutate=app/ --tests-dir=tests/

# Check results
mutmut results

# Fail if mutation score is below threshold
KILLED=$(mutmut results | grep -c "killed")
TOTAL=$(mutmut results | grep -c "")
SCORE=$((KILLED * 100 / TOTAL))

if [ "$SCORE" -lt 70 ]; then
    echo "GATE FAILED: Mutation score $SCORE% (threshold: 70%)"
    exit 1
fi
echo "GATE PASSED: Mutation score $SCORE%"

Note: Mutation testing is expensive (it runs your entire test suite for every mutation). Use it selectively:

• On critical modules (auth, payments, data integrity)
• As a nightly check, not on every PR
• On the specific tests generated by AI, not the entire suite