grep, curl, and jq

The Big Three for QA Engineers

Three commands handle the vast majority of QA investigation work on the command line: grep for searching through logs and files, curl for making HTTP requests, and jq for parsing JSON responses. Master these three and you can diagnose most issues from a terminal.

grep -- Search Logs and Files

grep searches for patterns in files and outputs matching lines. It is your first tool when investigating failures in log files.

Basic Usage

# Find ERROR lines in a log file
grep "ERROR" /var/log/app/application.log

# Case-insensitive search
grep -i "null pointer" /var/log/app/application.log

# Show 3 lines of context around each match (Before and After)
grep -C 3 "ERROR" /var/log/app/application.log

# Show only 3 lines before the match
grep -B 3 "ERROR" /var/log/app/application.log

# Show only 3 lines after the match
grep -A 3 "ERROR" /var/log/app/application.log

Searching Across Files

# Recursive search in all test files
grep -r "data-testid=\"checkout" tests/

# Recursive search with file name display
grep -rn "data-testid=\"checkout" tests/
# tests/checkout.spec.ts:15:  await page.locator('[data-testid="checkout"]').click()

# Search only specific file types
grep -r --include="*.ts" "waitForTimeout" tests/

# Exclude directories from search
grep -r --exclude-dir=node_modules "TODO" .

Counting and Filtering

# Count matching lines
grep -c "ERROR" /var/log/app/application.log
# Output: 47

# Count 500 errors in nginx logs
grep "HTTP 500" /var/log/nginx/access.log | wc -l

# Show only the matching part of the line (not the whole line)
grep -o "HTTP [0-9]\{3\}" /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Output:
#   12847 HTTP 200
#     234 HTTP 404
#      47 HTTP 500
#      12 HTTP 503

# Invert match: show lines that do NOT match
grep -v "healthcheck" /var/log/app/application.log
# Exclude noisy health check lines from log analysis

# Multiple patterns (OR logic)
grep -E "ERROR|FATAL|CRITICAL" /var/log/app/application.log

Regular Expressions with grep

# Match lines with timestamps in a specific range
grep -E "2024-01-15 14:3[0-9]:" /var/log/app/application.log

# Match email addresses
grep -oE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" users.txt

# Match IP addresses
grep -oE "([0-9]{1,3}\.){3}[0-9]{1,3}" access.log

curl -- Make HTTP Requests

curl sends HTTP requests from the command line. It is invaluable for quick API testing, health checks, and debugging network issues.

Basic Requests

# GET request
curl https://api.example.com/users

# Pretty-print JSON response (pipe through jq)
curl -s https://api.example.com/users | jq .

# POST with JSON body
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"name": "Test User", "email": "test@example.com"}'

# PUT request
curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name": "Updated User"}'

# DELETE request
curl -X DELETE https://api.example.com/users/123 \
  -H "Authorization: Bearer $TOKEN"

Inspecting Responses

# Response headers only (HEAD request)
curl -I https://api.example.com/users
# HTTP/2 200
# content-type: application/json
# cache-control: no-cache

# Verbose output (debug connection issues)
curl -v https://api.example.com/users
# Shows: DNS resolution, TCP connection, TLS handshake, request headers, response headers

# Show response headers along with body
curl -i https://api.example.com/users

# Only show the HTTP status code
curl -s -o /dev/null -w "%{http_code}" https://api.example.com/users
# Output: 200

Timing Requests

# Time the request (useful for performance testing)
curl -w "\nTime: %{time_total}s\nHTTP Code: %{http_code}\n" \
  -o /dev/null -s https://api.example.com/users
# Output:
# Time: 0.245s
# HTTP Code: 200

# Detailed timing breakdown
curl -w "\nDNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nFirst byte: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
  -o /dev/null -s https://api.example.com/users

Practical curl Scenarios

# Upload a file
curl -X POST https://api.example.com/upload \
  -F "file=@screenshot.png" \
  -H "Authorization: Bearer $TOKEN"

# Follow redirects
curl -L https://example.com/old-url

# Save response to a file
curl -o response.json https://api.example.com/users

# Send a request with cookies
curl -b "session=abc123" https://api.example.com/profile

# Set a timeout (prevent hanging)
curl --max-time 30 https://slow-api.example.com/endpoint

# Retry on failure
curl --retry 3 --retry-delay 5 https://flaky-api.example.com/endpoint

jq -- Parse JSON

jq is a command-line JSON processor. Most APIs return JSON, and jq lets you extract, filter, and transform that data.

Basic Usage

# Pretty-print JSON
curl -s https://api.example.com/users | jq .

# Extract a specific field
curl -s https://api.example.com/users/1 | jq '.name'
# Output: "John Doe"

# Extract a nested field
curl -s https://api.example.com/users/1 | jq '.address.city'
# Output: "New York"

# Extract from an array
curl -s https://api.example.com/users | jq '.[0].name'
# Output: "John Doe" (first user's name)

# Get all names from an array
curl -s https://api.example.com/users | jq '.[].name'
# Output:
# "John Doe"
# "Jane Smith"
# "Bob Wilson"

Filtering and Selecting

# Filter array elements
curl -s https://api.example.com/users | jq '.[] | select(.role == "admin")'

# Filter by numeric comparison
curl -s https://api.example.com/orders | jq '.[] | select(.total > 100)'

# Filter by string content
curl -s https://api.example.com/users | jq '.[] | select(.email | contains("@example.com"))'

# Multiple conditions
curl -s https://api.example.com/users | jq '.[] | select(.role == "admin" and .active == true)'

Transforming Data

# Count items in an array
curl -s https://api.example.com/users | jq length
# Output: 42

# Create a new object shape
curl -s https://api.example.com/users | jq '.[] | {name: .name, email: .email}'

# Extract specific fields into CSV-like format
curl -s https://api.example.com/users | jq -r '.[] | [.id, .name, .email] | @csv'
# Output:
# 1,"John Doe","john@example.com"
# 2,"Jane Smith","jane@example.com"

# Sort by field
curl -s https://api.example.com/users | jq 'sort_by(.created_at) | reverse'

# Group by field
curl -s https://api.example.com/users | jq 'group_by(.role) | .[] | {role: .[0].role, count: length}'

Working with Local Files

# Parse a local JSON file
jq '.dependencies' package.json

# Extract test configuration
jq '.scripts | to_entries | .[] | select(.key | startswith("test"))' package.json

# Modify JSON (useful for test data)
jq '.baseURL = "https://staging.example.com"' config.json > config-staging.json

Combining grep, curl, and jq

The real power comes from combining these tools:

# Check API health and alert on non-200
STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://api.example.com/health)
[ "$STATUS" -ne 200 ] && echo "API is down! Status: $STATUS"

# Find all admin users and count them
curl -s https://api.example.com/users | jq '[.[] | select(.role == "admin")] | length'

# Search API response for specific data
curl -s https://api.example.com/products | jq '.[] | select(.name | test(".*phone.*"; "i"))'

# Monitor endpoint response times in a loop
while true; do
  TIME=$(curl -s -o /dev/null -w "%{time_total}" https://api.example.com/health)
  echo "$(date): ${TIME}s"
  sleep 10
done

awk and sed (Quick Reference)

Two additional text processing tools that complement grep:

# awk: Calculate average response time from a log
awk '{sum += $4; count++} END {print sum/count "ms"}' response-times.log

# awk: Print specific columns
awk '{print $1, $4, $7}' /var/log/nginx/access.log

# sed: Replace staging URL with production URL
sed 's/staging.example.com/prod.example.com/g' test-config.json

# sed: Delete lines matching a pattern
sed '/healthcheck/d' application.log

# sed: Insert text at the beginning of a file
sed -i '1i # This is a test configuration file' config.json

ssh and scp

Remote server access is essential for QA work in non-local environments:

# Connect to a remote server
ssh qa@staging-server.example.com

# Run a command on a remote server without interactive session
ssh qa@staging-server.example.com "ps aux | grep node"

# Copy test results from remote to local
scp qa@staging-server.example.com:/var/log/test-results.xml ./

# Copy file from local to remote
scp test-data.json qa@staging-server.example.com:/tmp/

# SSH tunnel: Access a remote database locally
ssh -L 5432:localhost:5432 qa@staging-server.example.com
# Now connect to localhost:5432 to reach the remote Postgres

Hands-On Exercise

Use grep to find all ERROR lines in a log file, then count how many unique error messages there are
Use curl to make a GET and POST request to a public API (e.g., https://jsonplaceholder.typicode.com)
Use jq to extract and filter data from the API response
Combine all three: fetch a list of users with curl, parse with jq to find admins, and pipe through grep to find a specific email domain
Time an API request with curl -w and identify which phase (DNS, connect, TLS, first byte) is slowest
Write a one-liner that monitors an endpoint every 10 seconds and logs the status code and response time