observability

Observability — instrument for the question you'll ask at 2am

The [[debugging]] skill's first production move is "pull the real event." That move only works if someone instrumented the service before it broke — and that someone is you, now, while writing it. You can't debug what you didn't record, and you can't retrofit telemetry onto an incident that already happened. The bar: every service answers four questions without anyone SSHing anywhere — is it up, is it erroring, is it slow, and for whom?

How to apply this

Instrumentation is part of the feature, not a follow-up: a new endpoint ships with its metrics, error capture, and trace context wired, the same way it ships with tests. Run the self-check before calling any service change done.

Define "working" before instrumenting. Section 1's four questions are the per-operation floor every service owes; on top of them, write down 2–4 questions an on-call engineer will ask about this feature specifically ("is the retry actually recovering payments?", "which provider is slow?") — every signal you add beyond the floor must help answer one. Telemetry without a question is noise: if you can't name the questions, you'll log everything and learn nothing. The division of labor, as a rule of thumb: metrics tell you that something is wrong, traces tell you where, logs tell you why.

---

1. The four questions drive what you instrument

For every operation a service performs (each endpoint, each job, each consumer), emit enough to answer:

• Traffic — how often is this happening? (a counter)
• Errors — how often is it failing, split by kind? (a counter with a failure label)
• Latency — how long does it take, as a distribution? (a histogram — averages hide the slow tail where users actually live)
• Saturation — for resources (pools, queues, memory): how close to full? (a gauge)

If an operation emits these four, most incidents are diagnosable from a dashboard. If it emits none, every incident starts with "let's add logging and wait for it to happen again" — the most expensive sentence in operations.