data-pipeline-work

Data pipeline work

Where the failure mode is a wrong number, not an error.

Why this exists

Analytics and reporting are roughly half of large-organization IT, and pipelines fail in a way services don't: they keep running. A service that breaks throws errors, pages someone, and gets fixed within the hour. A pipeline that breaks produces plausible numbers that are wrong, and nobody notices until month-end — or until a regulator asks, or until a decision has already been made on them.

That single difference drives everything here. In service work you optimize for fast detection of loud failures. In pipeline work you optimize for detecting silence — building the checks that turn a quiet wrong answer into a loud one.

The second thing that catches service engineers out: pipelines are re-run. Re-running a service request is an edge case; re-running a pipeline is Tuesday. Idempotency is not a nice property here, it is the baseline requirement.

When this applies

• Analytics, ETL/ELT, warehouse, or reporting engagements
• A pipeline produces wrong or disputed numbers
• A job needs backfilling or reprocessing
• A report disagrees with the source system
• Adding a field that must flow through to reporting

When it doesn't

• Transactional service work — different failure modes entirely
• You need to understand the schema first — that's data-archaeology
• Live production incident — incident-triage, then come back

Prerequisites

• Locate the workspace: FDE_WORKSPACE, else the charter Location, else .fde/, else ../<repo>-fde/
• .fde/traces/data-model.md — the shape and eras of the source data
• .fde/02-system-map.md — where the pipeline sits
• Access to both source and destination, ideally, plus the orchestrator

Procedure

1. Establish the lineage end to end

Before changing anything, know the full path: source system → extraction → transformations → destination → the report someone actually reads.

For each hop record what it does, what it filters, what it joins, what it aggregates, and where the grain changes. Grain changes — one row per order becoming one row per customer per day — are where numbers stop reconciling and where almost every disputed figure originates.