reading-logs

Reading Logs

IRON LAW: Filter first, then read. Never open a large log file without narrowing it first.

Core Principles

1. Filter first - Search/filter before reading
2. Iterative narrowing - Start broad (severity), refine with patterns/time
3. Small context windows - Fetch 5-10 lines around matches, not entire files
4. Summaries over dumps - Present findings concisely, not raw output

Tool Strategy

1. Find Logs (Glob)

**/*.log
**/logs/**
**/*.log.*  # Rotated logs

2. Filter with Grep

# Severity search
grep -Ei "error|warn" app.log

# Exclude noise
grep -i "ERROR" app.log | grep -v "known-benign"

# Context around matches
grep -C 5 "ERROR" app.log  # 5 lines before/after

# Time window
grep "2025-12-04T11:" app.log | grep "ERROR"

# Count occurrences
grep -c "connection refused" app.log

3. Chain with Bash

# Recent only
tail -n 2000 app.log | grep -Ei "error"

# Top recurring
grep -i "ERROR" app.log | sort | uniq -c | sort -nr | head -20

4. Read Last

Only after narrowing with Grep. Use context flags (-C, -A, -B) to grab targeted chunks.

Investigation Workflows

Single Incident

1. Get time window, error text, correlation IDs
2. Find logs covering that time (Glob)
3. Time-window grep: grep "2025-12-04T11:" service.log | grep -i "timeout"
4. Trace by ID: grep "req-abc123" *.log
5. Expand context: grep -C 10 "req-abc123" app.log

Recurring Patterns

1. Filter by severity: grep -Ei "error|warn" app.log
2. Group and count: grep -i "ERROR" app.log | sort | uniq -c | sort -nr | head
3. Exclude known noise
4. Drill into top patterns with context

Red Flags

• Opening >10MB file without filtering
• Using Read before Grep
• Dumping raw output without summarizing
• Searching without time bounds on multi-day logs

Utility Scripts

For complex operations, use the scripts in scripts/:

# Aggregate errors by frequency (normalizes timestamps/IDs)
bash scripts/aggregate-errors.sh app.log "ERROR" 20

# Extract and group stack traces by type
bash scripts/extract-stack-traces.sh app.log "NullPointer"

# Parse JSON logs with jq filter
bash scripts/parse-json-logs.sh app.log 'select(.level == "error")'

# Show error distribution over time (hourly/minute buckets)
bash scripts/timeline.sh app.log "ERROR" hour

# Trace a request ID across multiple log files
bash scripts/trace-request.sh req-abc123 logs/

# Find slow operations by duration
bash scripts/slow-requests.sh app.log 1000 20