observability-audit

Observability Audit

Overview

If you can't observe it, you can't debug it. If you can't debug it, you can't fix it. Observability is the difference between "something is wrong" and "here's exactly what's wrong, where, since when, and for whom."

Core principle: Every production system must answer four questions at any point in time: What happened? When? Why? To whom?

The Iron Law

NO PRODUCTION SERVICE WITHOUT STRUCTURED LOGGING, HEALTH CHECKS, AND ERROR TRACKING. NO DEPLOYMENT WITHOUT MONITORING. NO ALERT WITHOUT A RUNBOOK.

When to Use

• "Can we debug production issues?"
• Before deploying a new service
• After a production incident (post-mortem revealed gaps)
• When investigation takes too long
• When errors are discovered by users instead of alerts
• During any codebase audit
• Before scaling a service

When NOT to Use

• Prototypes or throwaway experiments (basic console logging is fine)
• Static sites with no server component
• If you're just adding a single log line (use code-review instead)
• If the issue is specifically performance, not visibility (use performance-audit)

Anti-Shortcut Rules

YOU CANNOT:
- Say "logging is fine" — read actual log output and verify it's structured, contextual, and actionable
- Say "we have monitoring" — show the dashboard, verify it shows real-time data, check alerting rules
- Say "errors are tracked" — verify error grouping, notification routing, and triage workflow
- Skip checking what happens when a dependency goes down — simulate failure mentally or actually
- Assume health checks work — call the health endpoint and verify the response checks REAL dependencies
- Trust that alerts work — verify the notification pipeline end-to-end (alert → channel → person)
- Say "we'll add observability later" — it's needed BEFORE production, not after the first incident

Common Rationalizations (Don't Accept These)

Rationalization	Reality
"We'll add logging when something breaks"	You won't know it's broken without logging. Circular dependency.
"Console.log is fine for now"	Unstructured text logs are unsearchable, unfilterable, and fill up disk fast.
"We don't need distributed tracing yet"	You will need it the moment your first cross-service bug appears. Retrofitting is painful.
"Nobody looks at the dashboard"	Then your dashboard shows the wrong things. Fix the dashboard, don't eliminate monitoring.
"Alerts are too noisy so we ignore them"	Noisy alerts are worse than no alerts. Tune them — don't disable.
"Our error rate is low enough"	Low ≠ zero. One uncaught error in a payment flow is one too many.
"We know the system well enough"	You won't in 6 months, and neither will the person who replaces you.