metrics-instrumentation

Metrics Instrumentation

Normative spec for the backend half of operational observability: per-stage OpenTelemetry metrics in apps/opik-backend. The metrics are designed so a flow-ordered Grafana dashboard can read top-to-bottom — the failing stage is the one whose numbers break — with a per-workspace drill. Building that dashboard (layout, query contracts, per-customer name resolution, dependency panels, validation) is a separate concern, specified by the dashboard-authoring skill in comet monitoring tooling; this skill covers only what to emit.

Patterns applied in this implementation (online scoring is the worked example):

• Producer metrics — every stage that emits work is counted at the source: sampler decisions (sampler_decisions_total{decision}) and enqueue-to-Redis (enqueue_total{result}). A producer that emits nothing makes downstream starvation explainable rather than mysterious.
• Consumer metrics — throughput and per-stage timing on the side that drains the queue (processing_time, plus per-Redis-op read/claim/ack_and_remove/list_pending_time).
• Entrypoint RED — the workflow's front door (the HTTP ingest route) is measured Rate / Errors / Duration from http_server_request_duration_seconds, with 5xx broken down by endpoint × error_type × workspace, so an ingestion problem is never mistaken for a scoring problem.
• Errors — a dedicated error counter per stage, dimensioned by error_type (the exception class) and, for shared async plumbing, by component (listener/subscriber): processing_errors_total, unexpected_errors_total, and enqueue_total{result="error"} (a push failure = real loss).
• Success — success is derived, never double-counted: throughput − errors, surfaced as one "success rate" headline tile.
• Queue time & end-to-end latency — queue_delay (enqueue→pickup) is kept separate from processing_time (scorer/LLM work) so a backlog is distinguishable from a slow scorer; end-to-end = queue_delay + processing_time.
• Backpressure — poll-tick skips are counted but are benign (consumer busy), never lost work.
• Saturation & resource levels — gauges for in-flight work (max per pod) and JVM heap used-vs-limit per pod expose the pipeline approaching a ceiling before it starts failing (the USE method — utilization / saturation / errors — complementing RED).
• Volume & payload size — byte/char counters (bandwidth, total bytes) and payload-size distributions, broken down by content type and workspace, for cost and impact attribution.
• Per-workspace dimensioning — the customer drill (§1.3) is a first-class label, not an afterthought.
• Infrastructure dependencies — the datastores the flow leans on (Redis streams, ClickHouse, locks, MySQL) are surfaced on the dashboard from their exporters and system.query_log, so a "slow pipeline" resolves to the dependency causing it.

To see these conventions already in practice, grep apps/opik-backend for the existing metric families rather than specific classes (metric names are the stable contract; class locations move): the online-scoring *_sampler_decisions_total, *_enqueue_total{result}, *_processing_time_milliseconds, *_queue_delay_milliseconds, *_processing_errors_total{error_type} / *_unexpected_errors_total, the per-Redis-op *_{read,claim,ack_and_remove,list_pending}_time_milliseconds, and the attachment upload byte/size families.

Apply alongside the opik-backend skill (general conventions, logging rule).