log-analysis

Log analysis

Logs are a haystack that grows faster than you can read it. The skill is not reading logs — it is constructing a query narrow enough to answer one question, then widening only as far as needed.

The failure this prevents: scrolling. Scrolling through logs feels like work and finds only what happens to be near the cursor.

1. Ask one answerable question

Before opening anything, write the question down. "What went wrong?" is not answerable. These are:

• Did request abc-123 reach the payment service?
• How many 500s between 14:00 and 14:30, and on which endpoint?
• What is the first error after the deploy at 13:47?
• Which tenant accounts for the spike?

Done when: you have a question with a checkable answer.

2. Anchor on time and identity

Two anchors make everything else tractable:

• A time window: bound it tightly, then widen. Start a few minutes before the first known symptom, because the cause usually precedes it.
• An identifier: request ID, trace ID, user, order, tenant. One identifier that threads through services turns a search into a story.

If there is no correlating ID, that is your most important finding. Nothing else you do here will be reliable, and adding one should be the follow-up action.

Done when: you have a window and, ideally, an ID to follow.

3. Cut volume before reading

Filter, then aggregate, then read. Reading first is what wastes the afternoon.

# Shape of the problem before any individual line
grep ERROR app.log | awk '{print $5}' | sort | uniq -c | sort -rn | head

# Rate over time — is it constant, a spike, or a step change?
grep ERROR app.log | cut -c1-16 | uniq -c

# Follow one request across a file
grep 'req_id=abc-123' *.log | sort -k1,2

For structured logs, use the query language rather than grep — jq locally, or the platform's own filtering. Structured logs exist so you can aggregate; grepping them wastes that.

Done when: you know the shape — how many, how often, since when, affecting whom.