duckrun-projects

Building dbt projects with duckrun

duckrun is a dbt adapter where DuckDB executes your model SQL, delta-rs writes the result as a Delta table, and dbt orchestrates the DAG. It wraps dbt-duckdb, so everything dbt-duckdb gives you still works — view, seed, sources, tests, snapshots, Python models — and table / incremental write real Delta tables. The whole pipeline is pure Python: it runs identically on a laptop, in GitHub Actions, or in a Fabric Python notebook.

Mental model for every model you write: the SQL runs in an in-memory DuckDB; the result streams to delta-rs which commits it to <root_path>/<schema>/<model>; dbt then sees that table through a delta_scan view, which is how {{ this }}, ref() and is_incremental() resolve — including across separate dbt processes. The Delta tables are the ONLY state; there is no database file to manage.

Install and profile

pip install duckrun        # brings dbt-duckdb, duckdb, deltalake at the right pins

Do NOT separately pin duckdb or deltalake in your requirements — duckrun pins exact versions for documented upstream-bug reasons, and overriding them is the most common way to break a project.

# ~/.dbt/profiles.yml
my_project:
  target: dev
  outputs:
    dev:
      type: duckrun
      schema: dbo
      root_path: './warehouse'        # local path, or s3:// gs:// abfss://
      # storage_options: {}           # credentials, passed straight to deltalake

Things that differ from other adapters:

• threads: works, with a caveat. It's honored like any dbt adapter (default 1), but concurrent models share ONE DuckDB memory_limit — above 1 thread duckrun pins it once for the run at the tighter share and divides the delta-rs merge pool. So raise it for many small, network-bound models; leave it at 1 for a single big merge. A microbatch model's batches always run in order regardless (they all write one table).
• No database file. DuckDB is in-memory by default; don't add a path: expecting persistence — persistence is the Delta tables.
• Models land at <root_path>/<schema>/<model>; a per-model config(location=...) overrides that.