bauplan-dag-optimization

Optimize a Bauplan DAG

Contents

Mental model · Workflow · 1. Inventory · 2. Profile · 3. Materialization · 4. Inputs · 5. Model structure · 6. Project boundaries · 7. Partitioning · 8. Verify and measure · Proposal

References: partitioning.md for the profiling queries and partitioning heuristics, patterns.md for before/after code.

This skill uses the typed SDK (0.3.0+) syntax: inputs are Annotated[pa.Table, bauplan.Model(...)], projections are TableSchema classes passed as projection_schema=, and every model declares its output schema in its return annotation. If the project is on an older SDK, migrate it first with the bauplan-migrate-to-typed-sdk skill, or ask the user before optimizing old-style code.

Mental model

Bauplan handles I/O, Arrow conversion between models, where intermediate data lives and when work is scheduled. A model that is not materialized is still handed to its children by the platform; you do not manage memory or placement, and you should not try to. Performance comes from data modeling:

1. How much data each model reads: columns and rows.
2. How many tables are written to the lake and read back.
3. How the work is split into models and into projects.
4. How large tables are partitioned, so that reads prune and runs rewrite only what changed.

The biggest wins are almost always I/O: fewer materialized intermediates, narrower inputs, and partitioned tables that consumers can read slice by slice.

Platform facts that drive the decisions

1. Filters apply only to catalog inputs. filter= on bauplan.Model(...) works only on a table that already exists in the catalog when the run starts. Setting it on a model produced in the same run is rejected by the planner ("Apply the filter inside the function"). To read a slice of an intermediate, materialize it in one project and read it with filter= from a later project run.
2. Catalog filters and projections are pushed into the scan. The filter drives Iceberg file pruning (partition values and file statistics) before any data reaches the model, and only the projected columns are read. Pruning needs plain comparisons on columns: event_date >= '2025-01-01' prunes, LOWER(region) = 'emea' or other functions on the column do not. Parameters can be used inside filters with $name syntax, for example filter="event_date >= $start_date". The planner warns about unfiltered scans of tables of 20 GB or more.
3. Non-materialized intermediates are cheap. A model without a materialization_strategy (the default is NONE) does not write to object storage and no Iceberg commit happens. A materialized output is additionally written to object storage and committed to the catalog (REPLACE drops and rewrites the whole table).
4. Light models run in parallel. Within a job, when possible, independent models run in parallel. As a rule of thumb, the lighter the jobs, the easier it is to parallelize them. Heavier jobs, which take up a lot of memory, can result in an overall job slowdown.
5. Declarations are read statically. filter=, projection_schema=, the schema classes and the decorators are parsed from the source before execution, so filters must be string literals (a $name parameter inside the literal is fine), not f-strings or variables.