troubleshooting

Troubleshooting Skill

Expertise: Systematic debugging, log analysis, query profiling, memory/CPU profiling, regression tests.

Debugging Framework (RRCA)

1. REPRODUCE — make the bug happen reliably before touching code
2. REDUCE    — find the smallest input that triggers the bug
3. CAUSE     — identify the specific code line/condition responsible
4. ADDRESS   — fix + regression test + verify fix doesn't reappear

Never fix what you can't reproduce. A guess-and-check fix is technical debt.

Log Analysis Patterns

# Find all errors in last hour (structured logs with jq)
journalctl -u myapp --since "1 hour ago" | jq 'select(.level == "error")'

# Count errors by type
cat app.log | jq -r '.error_code' | sort | uniq -c | sort -rn | head -20

# Find slowest requests
cat access.log | jq 'select(.duration > 1000)' | jq -r '[.method, .path, .duration] | @csv'

# Trace a specific request by request_id
grep "request_id=req_abc123" app.log

# Find N+1 patterns: same query repeated many times in same request
grep "request_id=req_abc123" app.log | grep "db.query" | wc -l  # > 10 is suspicious

Database Query Debugging

-- Show currently running queries (PostgreSQL)
SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state
FROM pg_stat_activity
WHERE state != 'idle' AND query_start < now() - interval '5 seconds'
ORDER BY duration DESC;

-- Kill a blocking query
SELECT pg_terminate_backend(<pid>);

-- Find slow queries from pg_stat_statements
SELECT query, calls, mean_exec_time, total_exec_time
FROM pg_stat_statements
ORDER BY total_exec_time DESC LIMIT 10;

-- Check for table bloat (after many deletes/updates)
SELECT relname, n_dead_tup, n_live_tup,
       round(n_dead_tup * 100.0 / nullif(n_live_tup + n_dead_tup, 0), 2) AS dead_pct
FROM pg_stat_user_tables
WHERE n_dead_tup > 1000
ORDER BY dead_pct DESC;

Memory Leak Detection (Python)

# Detect growing memory with tracemalloc
import tracemalloc

tracemalloc.start()

# ... run suspected code ...

snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics("lineno")
for stat in top_stats[:10]:
    print(stat)  # shows file:line and allocated bytes

# Typical culprits:
# - Unbounded in-memory caches (dict that grows forever)
# - Event listeners not being removed
# - Circular references preventing GC