observability-slo

Observability and Reliability

The question this discipline answers is not "is the server up". It is:

When a user has a bad experience right now, can we tell — and can we find out why without deploying new code?

A system you cannot debug from its own output is a system you debug by guessing.

---

The three signals, and what each is for

Signal	Answers	Cost
Metrics	Is something wrong? Aggregate, cheap, alertable	Low, until cardinality explodes
Traces	Where is it wrong? One request across every service	Medium, sample it
Logs	Why is it wrong? Full detail for one event	High at volume

You need all three, and they must be linked: a metric spike leads to an exemplar trace, which leads to the log lines for that request. Three disconnected tools mean three separate investigations and a much longer time-to-diagnosis.

Use OpenTelemetry as the instrumentation layer — it keeps the vendor decision reversible, which matters because observability vendors are expensive and switching is otherwise a rewrite.

---

Structured logging

{"ts":"2026-08-24T16:18:12.451Z","level":"error","msg":"payment.capture_failed",
 "trace_id":"4bf92f...","span_id":"00f067...","request_id":"req_01HQ8",
 "user_id":"usr_01HQ8","org_id":"org_01HQ8","payment_id":"pay_01HQ8",
 "provider":"stripe","error_code":"card_declined","duration_ms":842}

Rules

1. JSON, not prose. Grep does not scale; structured queries do.
2. A correlation id on every line, generated at the edge (or accepted from the client), propagated through every service, job and outbound call. This single field is the difference between a two-minute investigation and a two-hour one.
3. Log events, not sentences. payment.capture_failed with fields beats "Failed to capture payment for user 123" — the first is queryable and aggregatable, the second is a string.
4. Levels with meaning. error = a human should look. warn = degraded but handled. info = business events worth keeping. debug = off in production. If everything is error, nothing is.
5. Never log secrets or personal data. Tokens, passwords, card numbers, full request bodies, authorization headers. Implement redaction as a layer in the logger, not as a habit at each call site. Logs are frequently the least protected copy of your data, and they are shipped to third parties.
6. Log at boundaries: request in/out, external call in/out, job start/end, state transitions. Not every line of business logic.
7. Retention bounded deliberately — a cost control and a privacy obligation.