bauplan-safe-ingestion

Safe Ingestion

Safely ingest data from S3 into the Bauplan lakehouse by isolating changes on a temporary branch, running quality checks, and only merging to main after validation succeeds. This pattern is formally known as Write-Audit-Publish (WAP) in the Iceberg ecosystem.

Implement this as a Python script using the bauplan SDK. Do NOT use CLI commands for the ingestion itself.

Environment Setup

Before writing the script, check whether the project uses uv (look for pyproject.toml or uv.lock). If so, use uv run python to execute scripts and uv add to install packages. Otherwise, use the system python and pip install.

Ensure the required packages are installed:

• bauplan (the Bauplan Python SDK — required)
• polars (if validation logic needs DataFrame operations — zero-copy Arrow interop)

Do not use pandas. Bauplan's client.query() returns a PyArrow table directly — you can access columns with result.column("name")[0].as_py() or convert to Polars with pl.from_arrow(result). No .to_arrow() call is needed. Pandas requires a full data copy and is slower.

The three phases:

1. Import — load data onto a temporary branch (never main)
2. Validate — run quality checks before publishing
3. Merge — promote to main only after validation passes

Branch safety: All operations happen on a temporary branch, NEVER on main. By default, branches are kept open for inspection after success or failure.

Atomic multi-table operations: merge_branch is atomic. You can create or modify multiple tables on a branch, and when you merge, either all changes apply to main or none do. This enables safe multi-table ingestion workflows.

Required User Input

Before writing the script, you MUST gather:

1. S3 path (required): The S3 URI pattern for the source data (e.g., s3://bucket/path/*.parquet)
2. Table name (required): The name for the target table
3. Validation path (required): See "Choosing a Validation Path" below
4. On success behavior (optional):
   • inspect (default): Keep the branch open for user inspection before merging
   • merge: Automatically merge to main and delete the branch
5. On failure behavior (optional):
   • inspect (default): Leave the branch open for inspection/debugging
   • delete: Delete the failed branch