orchestra-dbt-slim-ci-setup

Orchestra dbt Slim CI setup

Retrofit Slim CI onto an existing production dbt Orchestra pipeline (one pipeline for prod and CI). Generate or patch repo artifacts, validate pipeline YAML, and leave a verification checklist. Do not assume warehouse or Git connections can be created via API.

When to use

• User asks to set up Slim CI, dbt CI/CD in Orchestra, or run-pipeline for dbt PR checks.
• User asks what is missing from, or how to clean up, an existing dbt CI setup.
• Workspace is a dbt repo or user supplies dbt repo paths and pipeline YAML location.
• Default scope: retrofit an existing production pipeline and reuse its production dbt Core task; do not create a CI-only pipeline or a second dbt Core task for CI — artifact history is per task (see Guardrails).

Companion skill

After setup, failed PR checks or pipeline runs: start with identify-pipeline-error (SKILL.md) to find the failing task and cause; it routes dbt code/config issues to fix-pipeline-dbt-task (SKILL.md). Do not merge debug steps into this skill.

Workflow

1. Collect inputs — Follow references/inputs-matrix.md. Ask for must-haves; discover from repo when in a dbt project. Pipeline and task ids can come from the step 4 audit.
2. Detect context — In dbt repo: read dbt_project.yml, profiles.yml, orchestra/, .github/workflows/. Outside: require repo URL, pipeline YAML path, and where GHA lives.
3. Load Orchestra docs — Follow references/orchestra/mcp-playbook.md (Orchestra Documentation MCP, then Orchestra MCP).
4. Check for earlier attempts — Depth follows the ask (references/audit-existing-setup.md):
   • Implement (user names a pipeline and asks to set up Slim CI): quick conflict check only. No conflicts → carry on to step 5 without stopping. Conflicts → name them in a line each and ask only about those; the retrofit itself doesn't wait on the answer unless the conflict is the same pipeline.
   • Audit (user asks what's missing, to clean up or consolidate, or names no pipeline): full sweep, gap table and numbered fix plan, then stop for approval. Later steps run only for approved gaps; nothing to fix → skip to step 11.
5. Inventory pipeline — Read the keeper pipeline's YAML. Checklist: references/retrofit-checklist.md.
6. Patch pipeline YAML — Minimal changes from templates/pipeline-inputs-snippet.yml: dbt_branch, dbt_command (default = current prod command incl. --target ${{ ENV.DBT_TARGET }}), optional dbt_ci_schema_suffix passed to the task as DBT_CI_SCHEMA_SUFFIX, and max_active: ${{ ENV.DBT_PIPELINE_CONCURRENCY }} + on_limit: ${{ ENV.DBT_PIPELINE_ON_LIMIT }} so CI can run PRs in parallel while prod stays at 1 and merge-triggered deploys queue instead of being skipped. Git-backed: commit YAML; do not use update_pipeline. Orchestra-backed only: validate_pipeline then create_pipeline / update_pipeline.
7. Orchestra environments — CI and prod environments (names case-sensitive) with DBT_TARGET (ci / prod), DBT_PIPELINE_CONCURRENCY (e.g. 5 / 1) and DBT_PIPELINE_ON_LIMIT (QUEUE_LATEST / QUEUE; flag the cross-PR skip in references/retrofit-checklist.md before setting QUEUE_LATEST). Manual unless the environment MCP tools are available and the user approves.
8. dbt project — ci target alongside prod in the Orchestra dbt connection's profiles.yml (its schema is the per-PR prefix, e.g. ci → ci_123). Add or merge templates/dbt/generate_schema_name.sql and templates/dbt/drop_ci_schema.sql into macros/; if the project already overrides generate_schema_name, add only the ci branch. Projects with incremental models: add templates/dbt/selectors.yml. Bootstrap latest_production for that same dbt task (one successful run of it on the dbt repo default branch).
9. GitHub Actions — Add or patch templates/github-dbt-slim-ci.yml: PR job (CI env, cancel_on_exit: true, PR number as dbt_ci_schema_suffix), deploy job on merge (prod env, dbt_branch: main), and clean-up job dropping ci_<PR number> on close. Pin orchestra-hq/[email protected]+ (cancel_on_exit needs it). Incremental models: swap in the dbt_command values from templates/github-dbt-slim-ci-incremental.yml, after agreeing with the user whether CI starts from a dbt clone of production or from --defer alone — the two differ in test fidelity, run time, what ci_<PR number> ends up holding and what the ci role must reach (references/retrofit-checklist.md). Add --favor-state so a reverted model on a long-lived PR branch is read from production rather than from a stale CI relation. Ask how merges should deploy (rebuild modified, full refresh, or leave to the schedule) before keeping the deploy job. Separate pipeline repo: Action branch = pipeline YAML branch; dbt_branch only in run_inputs.
10. Secrets checklist — ORCHESTRA_API_KEY secret; pipeline id as a secret or vars.ORCHESTRA_DBT_PIPELINE_ID. Warehouse creds stay in the Orchestra dbt connection, so GitHub never needs them — including for schema clean-up.
11. Validate — validate_pipeline when definition available; dbt parse when possible. start_pipeline with Slim runInputs only with user approval.
12. Report — Use references/completion-report.md.