nexus-observability

Nexus Observability — API Failure Correlation Engine

Systematically correlate failures across distributed services to identify causal chains, blast radius, and the origin service — before proposing any fix.

---

Compatibility

• Supporting files: checklists/investigation-checklist.md, anti-patterns/common-mistakes.md, validation/output-validation.md
• Required tools: Read, Bash, Grep
• Optional tools: WebSearch (vendor-specific error lookups)
• Hands off to: nexus:debugging once origin service is identified; the roadmap-planner agent after fix is agreed upon

---

Core Principle

Never assign blame before building the full cross-service timeline. A cascade always has one origin — fixing a victim while the cause is live means the failure recurs.

---

Workflow

Step 1 — Context Acquisition

Collect before reading any logs. Require items 1–5 minimum; ask for all in one message:

#	Collect	Why
1	Verbatim error logs from all affected services	Paraphrased logs lose exact timestamps and error codes
2	Distributed traces (Jaeger/Zipkin/X-Ray/Tempo) for ≥3 failing requests	Shows exact call path and which hop introduced error/latency
3	Metrics: error rate, p99 latency, RPS, CPU/mem/conn-pool per service	Distinguishes saturation from errors from cascades
4	Exact first-elevated-error timestamp per service	Required for timeline in Step 2
5	Dependency map (direct + indirect call relationships)	Required for blast radius in Step 3
6	Changes in the 4h window: deploys, config, feature flags, cron, infra	Most failures are change-triggered
7	Alert firing history and order	Alert order mirrors propagation direction

---

Step 2 — Timeline Construction

Align all timestamps to UTC. Use first occurrence per service, not peak. Trust trace timestamps over log timestamps when they differ. Note data gaps explicitly — they are blind spots, not clean windows.

| Time (UTC) | Service     | Event                              | Error Rate |
|------------|-------------|------------------------------------|------------|
| T+00:00    | database    | connection pool at 95% utilization | 0%         |
| T+00:47    | database    | connection pool exhausted          | 0%         |
| T+01:03    | api-gateway | upstream timeouts begin            | 2%         |
| T+01:15    | auth-service| DB queries failing, JWT slow       | 8%         |
| T+01:22    | user-service| auth check timeouts                | 34%        |
| T+01:31    | api-gateway | circuit breaker opens on auth      | 78%        |