create-pipeline

Create a Calkit pipeline

Convert an existing repo with ad hoc scripts and manual steps into a fully reproducible Calkit pipeline. When complete, calkit run should reproduce all important outputs from scratch.

Step 1: Understand the existing repo

Before writing any YAML, map out what's already there. Start with the README. Users will typically write manual environment creation steps, and lists of script and commands to run in order. This is like a manual pipeline. Next:

1. List all scripts and notebooks: look in scripts/, notebooks/, src/, and the repo root for .py, .R, .jl, .m, .ipynb files.
2. Read each script to understand what it reads and writes.
3. Identify the dependency order: which outputs of script A become inputs to script B?
4. Note which environment each script needs (Python version, packages, R, Julia, MATLAB, Docker, etc.).

Ask the user if the order or dependencies are unclear. Do not guess at data flow.

Step 2: Initialize the project

If there is no calkit.yaml, run:

calkit init

This sets up Git (if needed) and DVC. If calkit.yaml already exists but has no pipeline section, skip this—you will add one.

Step 3: Define environments

Every stage must reference a named environment. Identify what's needed:

• Python: pyproject.toml, or no spec yet → uv, created with calkit new uv-env; an existing requirements.txt the project should keep → uv-venv; environment.yml → conda
• R: renv.lock → renv
• Julia: Project.toml → julia
• LaTeX: docker with texlive/texlive:latest-full
• MATLAB: matlab

Add environments to calkit.yaml:

environments:
  main:
    kind: uv
    path: pyproject.toml

Name a single Python environment main. With multiple environments, use descriptive names (e.g., analysis, paper).

Step 4: Try calkit xr first

For each script or notebook, try xr before writing YAML by hand. It auto-detects stage kind, environment, and I/O: