data-pipeline-cicd

Data Pipeline CI/CD

Purpose

Establish continuous integration and delivery for data pipelines covering SQL code quality, dbt model testing, schema change management, and environment promotion from dev to production.

Agent Protocol

Trigger

Exact user phrases: "data pipeline CI/CD", "dbt CI/CD", "SQLFluff", "SQL linting", "Atlantis SQL", "data pipeline testing", "environment promotion", "schema change management", "dbt test", "data CI", "database CI/CD", "data quality CI", "SQL review", "data deployment".

Input Context

Before activating, verify:

• Data transformation framework (dbt, SQLMesh, Dataform, custom SQL)
• Data warehouse (Snowflake, BigQuery, Redshift, Databricks, Postgres)
• CI platform (GitHub Actions, GitLab CI, CircleCI, Jenkins)
• Environments (dev, staging, prod) and promotion strategy
• Testing framework (dbt test, Great Expectations, data-diff, Soda)
• Schema migration tool (Alembic, Flyway, dbt migrations)

Output Artifact

CI/CD pipeline configuration with SQL linting rules, dbt test setup, environment promotion workflow, and schema change management as YAML, SQL, and configuration.

Response Format

# CI/CD pipeline configuration

# SQLFluff config

-- dbt test example

-- dbt Cloud job config

No preamble. No postamble. No explanations. No filler/hedging/transitions. Compress output.

Completion Criteria

• CI pipeline with SQLFluff linting configured
• dbt test suite with singular and generic tests
• Environment promotion workflow (dev -> staging -> prod) defined
• Schema change management with migration scripts
• Data diff or comparison testing in staging
• Rollback strategy documented

Max Response Length

4096

Workflow

CI Pipeline for dbt Projects

# .github/workflows/dbt-ci.yml
name: dbt CI
on:
  pull_request:
    branches: [main]
    paths: ['transform/**', 'profiles.yml']

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install sqlfluff==3.0.0 sqlfluff-templater-dbt
      - run: sqlfluff lint transform/models/ --dialect snowflake --templater dbt

  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install dbt-snowflake dbt-core==1.7.0
      - run: dbt deps --project-dir transform
      - run: dbt build --project-dir transform --target ci
      - run: dbt docs generate --project-dir transform