metrics-design

Metrics design

Metrics should answer two questions: "is it working?" and "if not, what's wrong?" Most teams emit 100 metrics and can't answer either.

Start with a framework

RED — for request-driven services

For every service endpoint / request type:

• Rate — requests per second.
• Errors — fraction of requests that failed.
• Duration — latency distribution (p50, p95, p99).

Three numbers. Covers 80% of "is the API working?"

USE — for resources

For every resource (CPU, disk, memory, thread pool, connection pool):

• Utilization — fraction of time busy.
• Saturation — queue depth / waiting work.
• Errors — errors encountered.

Answers "is something maxing out?"

The four golden signals (Google SRE)

• Latency — how long does a request take (success vs error separately).
• Traffic — how many requests.
• Errors — how many fail.
• Saturation — how full the system is.

Pick one framework per service and stick with it. Mixing creates dashboard chaos.

What a single metric needs

• A name — dot-separated or slash-separated, stable, meaningful. api.request.duration_ms, not timer_47.
• A unit — put it in the name (_ms, _bytes, _count). Avoids "is this seconds or milliseconds?" confusion.
• Labels / tags — for slicing. status, method, route, service. Be careful: each combination is a separate time series.
• A type — counter (monotonic), gauge (snapshot), histogram (distribution). Don't use a gauge where you need percentiles.

Label cardinality

This is the biggest metrics mistake. Each unique combination of label values creates a time series. Some labels are fine; some explode your storage.

• Fine: status (small enum), method (small), route (bounded list of endpoints).
• Dangerous: user_id, request_id, ip, session_id, full URL with query strings.

Rule of thumb: a label's cardinality should be bounded by a small constant (≤~100) or by the cluster topology (not by user count).