reproducible-pipelines

Reproducible Pipelines

Reference for building reproducible research pipelines: from project directory structure to automated workflows to journal-ready replication packages. Every computational result should be regenerable from raw data by running a single command.

When to Use This Skill

Use when the user is:

• Setting up a new empirical research project
• Building or debugging a Makefile/Snakemake/DVC pipeline
• Preparing a replication package for journal submission
• Managing computational environments (conda, Docker, renv)
• Tracking data provenance or versioning large datasets
• Debugging "works on my machine" reproducibility failures

Skip when:

• The task is about estimation methodology (use causal-inference or structural-modeling skill)
• The task is git workflow management (see workflows-work/references/worktree-patterns.md)
• The task is about orchestrating Claude agents (see slfg/references/orchestration-patterns.md)

Where to Start

• New project? Start with Directory Structure below
• Adding a workflow manager? Jump to Workflow Managers (Make / Snakemake / DVC)
• Preparing for submission? Jump to Pre-Submission Checklist

Project Directory Structure

Use a standardized layout from the start. This is the structure expected by most replication reviewers:

project/
├── README.md                 # Master documentation (how to replicate)
├── Makefile                  # Or Snakefile — single entry point
├── environment.yml           # Conda environment (or requirements.txt)
├── data/
│   ├── raw/                  # Original, immutable data files
│   │   └── README.md         # Data sources, access instructions, citations
│   ├── intermediate/         # Cleaned/transformed data (gitignored, regenerable)
│   └── final/                # Analysis-ready datasets (gitignored, regenerable)
├── code/
│   ├── 01_clean.py           # Data cleaning
│   ├── 02_build.py           # Variable construction, merges
│   ├── 03_estimate.py        # Main estimation
│   ├── 04_robustness.py      # Robustness checks
│   └── 05_tables_figures.py  # Output generation
├── output/
│   ├── tables/               # LaTeX/CSV tables (gitignored, regenerable)
│   └── figures/              # PDF/PNG figures (gitignored, regenerable)
├── docs/
│   ├── brainstorms/          # Research brainstorming docs
│   ├── plans/                # Implementation plans
│   └── codebook.md           # Variable definitions
├── tests/                    # Validation tests
│   ├── test_clean.py
│   └── test_estimates.py
└── paper/
    └── manuscript.tex        # The paper itself