5.5Trace-Based Testing

A powerful pattern is writing assertions against traces -- verifying not just that a request succeeded, but that it followed the expected path through the system:

# test_trace_assertions.py
import requests
import time

def test_order_flow_creates_expected_trace(trace_client):
    """Verify the order flow hits all expected services in the correct order."""
    # Trigger the order flow
    response = requests.post("https://api.example.com/orders", json={
        "items": [{"sku": "LAPTOP-1", "qty": 1}],
        "payment": {"method": "card", "token": "tok_test_123"},
    })
    assert response.status_code == 201
    trace_id = response.headers["X-Trace-Id"]

    # Allow time for spans to propagate to the backend
    time.sleep(5)
    trace_data = trace_client.fetch_trace(trace_id)

    # Assert all expected services are present
    service_names = [span["service_name"] for span in trace_data["spans"]]
    assert "api-gateway" in service_names
    assert "order-service" in service_names
    assert "payment-service" in service_names
    assert "inventory-service" in service_names

    # Assert ordering: payment happens before inventory reservation
    payment_span = next(s for s in trace_data["spans"]
                       if s["operation"] == "charge_payment")
    inventory_span = next(s for s in trace_data["spans"]
                         if s["operation"] == "reserve_inventory")
    assert payment_span["end_time"] <= inventory_span["start_time"]

    # Assert performance: total trace duration under 3 seconds
    root_span = next(s for s in trace_data["spans"] if s["parent_id"] is None)
    assert root_span["duration_ms"] < 3000

    # Assert no error spans
    error_spans = [s for s in trace_data["spans"] if s.get("status") == "ERROR"]
    assert len(error_spans) == 0, f"Unexpected errors in trace: {error_spans}"

What Trace-Based Tests Catch

Assertion	What It Catches
Service present in trace	Missing service call (regression in integration)
Span ordering	Race conditions, incorrect orchestration
Total trace duration	End-to-end performance regression
No error spans	Swallowed errors, silent failures
Expected attributes on spans	Missing context propagation
Span count	Unexpected service calls (N+1 queries, extra retries)