observability-and-instrumentation

Observability and Instrumentation

Overview

Code you can't observe is code you can't operate. Instrumentation is not a post-launch add-on — it's written alongside the feature, the same way tests are.

When to Use

• Building any feature that will run in production
• Adding a new service, endpoint, background job, or external integration
• A production incident took too long to diagnose
• Setting up or reviewing alerting rules

Process

1. Define "working" before instrumenting

Write down 2–4 questions an on-call engineer will ask about this feature:

FEATURE: checkout payment retry
QUESTIONS ON-CALL WILL ASK:
1. What fraction of payments succeed on first attempt vs after retry?
2. When a payment fails permanently, why?
3. Is the payment provider slower than usual?
→ Every signal below must help answer one of these.

2. Pick the right signal

Signal	Answers	Cost
Structured log	"What happened in this specific case?"	Per-event
Metric	"How often / how fast, in aggregate?"	Fixed per series
Trace	"Where did time go across services?"	Per-request, sampled

Rule: metrics tell you that something is wrong, traces tell you where, logs tell you why.

3. Structured logging

Log events, not prose. Every log line is a JSON object with a stable event name:

// BAD: logger.info(`Payment ${id} failed for user ${userId}`);
// GOOD:
logger.warn({
  event: 'payment_failed',
  paymentId: id,
  provider: 'stripe',
  errorCode: err.code,
  attempt: n,
}, 'payment failed');

Log levels:

Level	Meaning	On-call action
error	Invariant broken	Investigate
warn	Degraded but handled	Watch for trends
info	Significant business event	None
debug	Diagnostic detail	Off in production

Correlation IDs are mandatory. Generate/accept a request ID at the system boundary and attach it to every log line, span, and outbound call.