airflow-dynamic-task-mapping-to-orchestra

Airflow Dynamic Task Mapping → Orchestra Matrix Task Groups

Overview

Airflow's dynamic task mapping (task.expand(), introduced 2.3+) creates N copies of a task at runtime from a list or dict — one task instance per file in a directory, per page of an API response, per row of a config table. The mapped values can be known statically at DAG-parse time (expand(param=["a", "b", "c"])) or come from an upstream task's return value via XComArg (expand(param=some_task.output)), in which case the fan-out count isn't known until that upstream task actually runs.

Orchestra's equivalent is a matrix task group: a TaskGroupModel (an entry under pipeline:) with a matrix: block. Each value in the matrix's input list creates one parallel execution of that entire stage — so a stage with a matrix: block behaves like Airflow's mapped-task fan-out, except the unit that repeats is the whole stage, not an individual task. Reference the current iteration's value in any task's parameters: via ${{ matrix.<input_name> }}.

Confirmed vs. inferred: the matrix/MatrixBlockModel schema below was fetched directly from Orchestra's live JSONSchema on 2026-07-13 and is accurate as of that date. Unlike most of this skill set, though, there's no real customer DAG or /api/convert test run behind this skill yet — treat generated matrix YAML as needing extra review before trusting it in production (see the last Gotcha).

MatrixBlockModel (confirmed schema)

pipeline:
  <stage-id>:
    matrix:
      inputs:
        <input_name>: [value1, value2, value3]   # a static list, OR a string expression resolving to a list
      max_parallel: 4          # optional — cap concurrent iterations
      sequential: false        # optional — run iterations one at a time instead of in parallel
      continue_on_error: false # optional — keep running other iterations if one fails
    tasks:
      <task-id>:
        parameters:
          some_param: ${{ matrix.<input_name> }}   # reference this iteration's value
        ...