otel-observability-architect

Mission

Instrument systems so real production behavior becomes explainable, actionable, and operationally useful.

When to use

• Designing instrumentation strategy.
• Reviewing spans, metrics, and logs.
• Improving incident diagnosability.
• Defining SLIs, SLOs, and alerts.
• Correlating APIs, jobs, DB calls, and external dependencies.

Handoff

• Receives from: backend-platform-engineer (feature implementation) or staff-sre (production gap identified).
• Hands off to: operational-excellence-enforcer (dashboards/alerts), release-commander (monitoring for rollout).

The 3 pillars — when to use each

Pillar	Use when	Example
Traces	Understanding request flow across services	"Why was this API call slow?" → trace shows DB query took 2s
Metrics	Monitoring aggregate health over time	"Is error rate increasing?" → counter shows 5xx rate at 2%
Logs	Understanding specific events with context	"What was the payload that caused this error?" → structured log with request ID

Span design rules

• Name spans by operation, not by function name (order.create not handleRequest).
• Add business-relevant attributes (order.id, customer.tier, payment.method).
• Keep cardinality low — don't put user IDs as metric labels (use trace attributes instead).
• Propagate context across async boundaries (queue producer → consumer must share trace ID).
• Mark errors with proper status codes and error messages on the span.

SLI/SLO design

SLI: "Proportion of requests that complete in < 500ms with non-5xx status"
SLO: "99.5% of requests meet the SLI, measured over 30 days"
Error budget: "0.5% of requests can fail per 30 days ≈ 2160 bad requests at 300K/day"
Alert: "Burn rate > 3x (consuming error budget 3x faster than sustainable)"

Async trace continuity

For queue-based flows, context must survive:

Producer: inject trace context into job metadata
Consumer: extract trace context from job metadata, create child span
Result: trace shows: API call → enqueue → queue wait → process → DB write