airflow-xcoms-to-orchestra

Airflow XComs → Orchestra Outputs

Overview

Airflow XComs (cross-communications) let tasks share small values by pushing to and pulling from a key-value store. Orchestra replaces this with a typed outputs system: a task explicitly sets named outputs using the Orchestra SDK, and downstream tasks or conditions reference them via ${{ ORCHESTRA.PIPELINE_RUN_TASKS['task_id'].OUTPUTS['key'] }}.

Key architectural difference: XComs are implicit (any task can pull from any other). Orchestra outputs are explicit — a task must opt in with set_outputs: true in its parameters and call the Orchestra SDK's set_output() on an instantiated client, not a bare imported function — see the code below.

---

Pattern Mapping

Airflow pattern	Orchestra equivalent
ti.xcom_push(key='count', value=42)	OrchestraSDK client's client.set_output('count', 42) — see setup below
ti.xcom_pull(task_ids='my_task', key='count')	${{ ORCHESTRA.PIPELINE_RUN_TASKS['my-task'].OUTPUTS['count'] }}
@task return value	client.set_output('return_value', result)
XCom-driven BranchPythonOperator	condition: expression on downstream stage
Passing large datasets via XCom	Restructure: write to S3/Snowflake, pass only the path/table name as output

---

Setting Outputs in a Python Task

The task script must instantiate the OrchestraSDK client and call .set_output() on it — there is no bare set_output() function to import. Enable output capture with set_outputs: true in parameters; Orchestra auto-injects ORCHESTRA_API_KEY into every PYTHON_EXECUTE_SCRIPT task's environment, so the script doesn't need a connection or secret for this.

stage-extract:
  tasks:
    get-row-count:
      integration: PYTHON
      integration_job: PYTHON_EXECUTE_SCRIPT
      name: get_row_count
      connection: my_python_conn_12345
      parameters:
        command: 'python scripts/get_row_count.py'
        python_version: '3.12'
        set_outputs: true          # required — enables output capture
      depends_on: []