bauplan-data-pipeline

Creating a New Bauplan Data Pipeline

This skill guides you through creating a new bauplan data pipeline project from scratch, including the project configuration and transformation models.

CRITICAL: Branch Safety

NEVER run pipelines on main branch. ALWAYS use a separate data branch for development.

Branch naming convention: <username>.<branch_name> (e.g., john.feature-pipeline). Get your username by running bauplan info.

Environment Setup

Before writing pipeline code, check whether the project uses uv (look for pyproject.toml or uv.lock). If so, use uv run to execute commands and uv add to install packages. Otherwise, use pip install.

Ensure the environment has a typed SDK build (0.3.0+). Typed declarations are only available with SDK 0.3.0+: older versions are not compatible. Use that same environment for all CLI commands and verify connectivity with bauplan info. Models declare their own runtime dependencies via @bauplan.python('3.11', pip={...}); annotation imports require pyarrow locally too.

Table References

• Source tables must already exist in the bauplan lakehouse before building a pipeline.
• Verify source tables exist and understand their schema before writing any code.
• Always use fully-qualified names: <namespace>.<table_name> (e.g., bauplan.taxi_fhvhv).
• Default namespace: bauplan.

Required User Input

Before writing a pipeline, you MUST gather the following from the user:

1. Pipeline purpose (required): What transformations should the DAG perform? What is the business logic or goal?
2. Source tables (required): Which tables from the lakehouse should be used as inputs? Verify they exist with bauplan table get <namespace>.<table_name>.
3. Output tables (required): Which tables should be materialized as final outputs?
4. Materialization strategy (optional, default: REPLACE): Should output tables use REPLACE or APPEND?
5. Strict mode (optional, default: on): Should failing expectations be allowed to pass, which needs --no-strict?