derivation-aggregate-metrics

derivation-aggregate-metrics

Stateless Estuary derivation that continuously aggregates source documents into metrics, using schema-level reduction annotations to combine documents with the same key.

Prereq: read derivation-basics first for concepts, project layout, workflow, and the stateless-vs-stateful distinction.

Docs:

• https://docs.estuary.dev/getting-started/tutorials/continuous-materialized-view/ — end-to-end tutorial with reductions
• https://docs.estuary.dev/reference/reduction-strategies/ — reference for sum, merge, minimize, maximize, append, firstWriteWins, lastWriteWins

When to use this over alternatives

• Daily / hourly / per-customer aggregations: sum orders by day, count events per user, etc.
• Running totals: lifetime revenue per customer that updates in real time
• Statistical metrics per group: min / max / sum / count per sensor, per region
• Continuous materialised views: stream-native replacement for scheduled aggregation queries

Reach for other skills when:

• Output is 1:1 with input (no grouping) → derivation-filter-transform
• Output is many-per-input (not grouping) → derivation-flatten-array
• Combining across multiple source collections on a shared key → derivation-join-collections (which uses the same reduction mechanics)
• Custom state you update procedurally (not pure reduction) → derivation-stateful-logic

How it works (one paragraph)

You don't write aggregation logic yourself. Your lambda emits one "delta" document per source input (1 AS order_count, $total AS revenue). The derived collection's schema carries reduce: annotations telling Estuary's runtime how to merge documents with the same key (sum adds numeric values, maximize keeps the latest timestamp, etc.). At materialisation time, the runtime reduces all deltas with the same key into a single final document. This is why shuffle: any works even though the effect is aggregation — the lambda itself is stateless.