observability-and-instrumentation

Observability and Instrumentation

Overview

Code you can't observe is code you can't operate. When a request hangs, a queue backs up, or a TaskCanceledException spikes at 2am, the only thing standing between you and a multi-hour investigation is the telemetry you added while building the feature — not after.

Instrumentation is a development responsibility, not a post-launch chore. .NET gives you first-class, vendor-neutral building blocks for all three signals — ILogger for structured logs, System.Diagnostics.Metrics for metrics, and System.Diagnostics.Activity + OpenTelemetry for traces — so there's no reason to defer it.

This skill is the bridge between debugging-and-error-recovery (dev-time diagnosis) and shipping-and-launch (launch-day monitoring): instrument as you build so production behavior is visible before you need it.

When to Use

• Shipping a feature that will run in production
• Adding a new service, background worker, or external integration (HTTP, EF Core, message queue)
• A production issue was reported but the existing data can't explain what happened
• Setting up or revising alerts
• Reviewing a PR that introduces I/O, retries, caching, or fan-out

Process

1. Start from the questions, not the tools

Before instrumenting, write down the questions an on-call engineer will ask at 2am: Is it the database or the API? Which tenant? Is it one endpoint or all of them? Did the retry storm start before or after the deploy? Instrument to answer those questions — not to "add logging."

2. Match the signal to the question

• Logs answer "why" — the specific event, with context. Use ILogger.
• Metrics answer "how often / how much" — aggregates over time. Use System.Diagnostics.Metrics.
• Traces answer "where" — which hop in a distributed call chain. Use Activity / OpenTelemetry.

3. Structured logging with ILogger