pipeline

Data Workflow Specification

Generate a data pipeline specification — a structured description of how data flows through transformations, what quality invariants must hold, and which concepts own the source and output data.

How to interpret $ARGUMENTS

Determine the mode from the argument:

• Path to directory/file (e.g. src/lib/server/syncs/): Retrofit mode — read the existing data transformation code, identify the pipeline stages, and propose a PIPELINE.md spec. Flag any quality invariants that are assumed but not checked.
• Pipeline description (e.g. Sentiment scoring with recency weighting): Greenfield mode — design a data pipeline spec from the description. Define sources, stages, outputs, and invariants.
• No arguments: Discovery mode — analyze the project for data workflows (queries, aggregations, sync chains, batch operations) and list candidates for PIPELINE.md specs. Do NOT generate full specs; ask the user which to elaborate. If discovery surfaces cross-concept coordination patterns (event handlers, scheduled jobs) rather than data transformations, note that /wyx:sync covers those.

PIPELINE.md Format

Write the spec as a PIPELINE.md file placed next to the data transformation code (one per directory).

# pipeline: [Name]

## purpose
[Single sentence: what data this pipeline produces and for whom]

## sources
- [name]: [table/file/API] → [key fields] ([row estimate or "unbounded"])

## stages

### [stage-name] [tool: <tool-name>]
in: [source or previous stage output]
out: [what this stage produces]
[1-3 lines: transformation logic]
quality: [invariant for this stage]

### [stage-name] [tool]
in: [...]
out: [...]
[...]
quality: [invariant]

## outputs
- [name]: [table/format] → [key fields] ([row estimate])

## invariants
- [data quality rules that must always hold across the full pipeline]
- [e.g. "output rows <= input rows" or "scores in [-1.0, 1.0]"]
- [e.g. "no null values in amount_tax_excluded after stage 2"]

## triggers
- [what causes this pipeline to run: sync event, API call, schedule, manual]

## data boundary
- [which concept owns the source data — read through its service, not direct SQL]
- [which concept owns the output data — write through its service, not direct INSERT]