3.2RCA Frameworks

The 5 Whys

The simplest and most widely used RCA technique. Start with the problem and ask "why" repeatedly until you reach a root cause that is systemic, not symptomatic.

Example: Production outage due to database connection exhaustion

Level	Question	Answer
Problem	Why did the application go down?	The database connection pool was exhausted
Why 1	Why was the connection pool exhausted?	A slow query was holding connections for 30+ seconds
Why 2	Why was the query slow?	The query was doing a full table scan on a 50M-row table
Why 3	Why was there a full table scan?	The query was missing an index on the created_at column
Why 4	Why was the index missing?	The migration that should have added the index failed silently
Why 5	Why did the migration fail silently?	Our migration runner does not alert on failures; it logs to a file nobody monitors

Root cause: Migration failures are not monitored or alerted. Contributing cause: No performance testing catches slow queries before production.

Corrective actions:

1. Add alerting for failed migrations (prevents this class of issue)
2. Add the missing index (fixes this specific issue)
3. Add query performance testing to CI pipeline (catches slow queries earlier)

Common mistakes with 5 Whys:

Mistake	Problem	Fix
Stopping too early	"The query was slow" is a symptom, not a root cause	Keep asking why until you reach a process or system failure
Only one chain	Complex incidents have multiple contributing causes	Ask "why" from multiple starting points
Landing on a person	"Because Alice did not add the index" blames a person, not a system	Ask why the system allowed that to happen
Too abstract	"Because our process is bad" is too vague to act on	Be specific: what process, what gap, what change

Fishbone Diagram (Ishikawa)

The fishbone diagram organizes potential causes into categories, which is useful for complex incidents with multiple contributing factors.

                           ┌── Process ──────┐
                           │  No migration    │
                           │  monitoring      │
                           │                  │
              ┌── People ──┤   ┌── Tools ────┤
              │  No query  │   │  No slow     │
              │  review    │   │  query       │
              │  process   │   │  detection   │
              │            │   │              │
Problem: ─────┤            ├───┤              ├───→ Database
Database      │            │   │              │     Outage
Outage        │            │   │              │
              │            │   │              │
              └── Env ─────┤   └── Testing ──┤
                 Staging DB│      No load     │
                 has 1K    │      testing     │
                 rows, not │      with        │
                 50M       │      production  │
                           │      data        │
                           │      volumes     │
                           └─────────────────┘

Standard fishbone categories (the 6 Ms):

• Methods: Processes, procedures, policies
• Machines: Tools, infrastructure, environments
• Materials: Data, inputs, dependencies
• Measurements: Monitoring, alerting, metrics
• Manpower: Skills, training, staffing
• Mother Nature: External factors, third-party services

Fault Tree Analysis

Fault tree analysis works backward from the failure using Boolean logic (AND/OR gates) to identify all possible cause combinations.

                    Database Outage
                          |
                         AND
                    /           \
          Slow Query          No Connection
          Exists              Pool Recovery
            |                      |
           OR                     AND
         /    \              /          \
   Missing   Unoptimized   No pool     No timeout
   index     query plan    monitoring  configured

When to use fault tree analysis: Complex, safety-critical systems where you need to understand all possible failure paths. Common in aviation, medical devices, and nuclear systems. Less common in web applications, but valuable for critical infrastructure.