airflow-sensors-to-orchestra

Airflow Sensors → Orchestra Sensors Block

Overview

Airflow sensors are tasks that poll an external system until a condition is met, then allow downstream tasks to proceed. In Orchestra, sensors are pipeline triggers — they live in the sensors: block at the pipeline root (not inside pipeline:). When all sensor checks pass, the pipeline run is triggered automatically.

Key difference: Airflow sensors are inline DAG steps. Orchestra sensors are external watchers that start pipeline runs. They use a cron window + polling interval model.

Sensor checks also carry a connection: field — for mapping the sensor's aws_conn_id=/conn_id= to the right Orchestra connection type, see airflow-connections-to-orchestra.

SensorModel Structure

sensors:
  <sensor-id>:
    name: My Sensor                  # required, max 100 chars
    cron: '0 8 * * ? *'               # required — when the check window opens
    timezone: UTC                    # required — IANA timezone
    timeout_mins: 60                 # required — max 7200; must be < cron interval
    frequency_secs: 60               # optional — polling interval (60–600, default 60)
    exclude: []                      # optional — YYYY-MM-DD dates to skip
    run_inputs: {}                   # optional — inputs to pass when sensor triggers

    checks:                          # required — dict of SensorCheckModel
      <check-id>:
        integration: SNOWFLAKE       # IntegrationsEnum
        sensor_type: SNOWFLAKE_QUERY # SensorChecksEnum (see table below)
        connection: my_snowflake_12345
        parameters:
          query: "SELECT COUNT(*) FROM daily_files WHERE date = CURRENT_DATE"
        map_outputs:                 # optional — pipe check results to pipeline inputs
          file_count: "result"       # pipeline input name → check output field

    alerts:                          # optional — sensor-level alerts
      - name: sensor-timed-out
        statuses: [FAILED]
        destinations:
          - integration: SLACK
            destination: '#data-alerts'