4.2Pipes

The pipe operator | takes the standard output (stdout) of the left command and passes it as standard input (stdin) to the right command.

Basic Pipe Examples

# Count how many POST requests returned 500
grep "POST /api" /var/log/nginx/access.log | grep "500" | wc -l

# Find the 10 most frequent error messages
grep "ERROR" app.log | sort | uniq -c | sort -rn | head -10

# Show unique IP addresses accessing your server
awk '{print $1}' /var/log/nginx/access.log | sort -u | wc -l

# Find the largest files in a directory
du -sh * | sort -rh | head -5

Breaking Down a Complex Pipe

Let us trace through the "10 most frequent error messages" example step by step:

# Step 1: Get all ERROR lines
grep "ERROR" app.log
# Output: hundreds of error lines

# Step 2: Sort them (required for uniq)
grep "ERROR" app.log | sort
# Output: error lines in alphabetical order

# Step 3: Count consecutive duplicates
grep "ERROR" app.log | sort | uniq -c
# Output:
#   47 ERROR: Connection timeout to payment-service
#   23 ERROR: Invalid session token
#   12 ERROR: Database query exceeded timeout

# Step 4: Sort by count (descending, numeric)
grep "ERROR" app.log | sort | uniq -c | sort -rn
# Output: Most frequent errors first

# Step 5: Show only the top 10
grep "ERROR" app.log | sort | uniq -c | sort -rn | head -10
# Output: The 10 most frequent error messages with counts