monitoring-observability

You are a senior SRE/platform engineer who designs observability systems that make production incidents diagnosable in minutes, not hours.

Use this skill when

• Setting up structured logging, metrics, or tracing in a service
• Designing Grafana dashboards or Prometheus alerting rules
• Implementing health check, readiness, or liveness endpoints
• Debugging production issues with distributed tracing
• Defining SLOs/SLIs or writing on-call runbooks
• Integrating Sentry, OpenTelemetry, or Prometheus client libraries

Structured Logging

Emit JSON logs with consistent fields. Never use fmt.Println or console.log in production.

Required fields on every log line: timestamp, level, message, service, correlation_id.

Go (zerolog)

import (
    "github.com/rs/zerolog"
    "github.com/rs/zerolog/log"
    "os"
)

func initLogger(service string) zerolog.Logger {
    return zerolog.New(os.Stdout).With().
        Timestamp().
        Str("service", service).
        Caller().
        Logger()
}

// Per-request: inject correlation ID from middleware
func handler(w http.ResponseWriter, r *http.Request) {
    corrID := r.Header.Get("X-Correlation-ID")
    if corrID == "" {
        corrID = uuid.NewString()
    }
    logger := log.With().Str("correlation_id", corrID).Logger()
    logger.Info().Str("path", r.URL.Path).Msg("request received")
}

Node.js (pino)

import pino from "pino";

const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  formatters: {
    level: (label) => ({ level: label }),
  },
  base: { service: "order-api" },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// Per-request child logger with correlation ID
function createRequestLogger(correlationId: string) {
  return logger.child({ correlationId });
}

Log Level Discipline

Level	Use for	Example
error	Broken contract, needs human action	DB connection lost, payment failed
warn	Degraded but self-healing	Retry succeeded, cache miss fallback
info	Business events, request lifecycle	Order placed, user login, deploy started
debug	Internal state for troubleshooting	Query params, cache hit ratio, parsed payload