airflow-testing-to-orchestra

Airflow Testing Operators → Orchestra Test Jobs

Overview

Airflow data quality checks are typically SqlCheckOperator (pass/fail based on SQL result), SQLThresholdCheckOperator (pass/fail with min/max bounds), or framework operators like GreatExpectationsOperator and SodaCheckOperator. Orchestra has native test job types for SQL-based checks across all major warehouses, with configurable error and warning thresholds.

For mapping the conn_id= on these check operators to the right Orchestra connection, see airflow-connections-to-orchestra.

---

Test Job Pattern

All Orchestra test integrations follow the same parameter pattern:

parameters:
  statement: "SELECT COUNT(*) FROM orders WHERE amount < 0"
  error_threshold_expression: "> 0"    # FAIL if any negative amounts
  warn_threshold_expression: "> 0"     # WARN at same threshold (or set looser)

The statement must return a single numeric value (count, sum, percentage). The result is compared against the threshold expressions. Both thresholds default to "> 0" — fail if any rows returned.

---

Available Test Integration Jobs

Integration	integration_job	Warehouse
SNOWFLAKE	SNOWFLAKE_RUN_TEST	Snowflake
GCP_BIG_QUERY	GCP_BQ_RUN_TEST	BigQuery
POSTGRES	POSTGRES_RUN_TEST	PostgreSQL
DATABRICKS	DATABRICKS_RUN_TEST	Databricks SQL
FABRIC_SYNAPSE	FABRIC_SYNAPSE_RUN_DQ_TEST	Microsoft Fabric
SNOWFLAKE	SNOWFLAKE_SCHEMA_VALIDATION	Column type/constraint checks
SNOWFLAKE	SNOWFLAKE_ANOMALY_DETECTION	ML-based anomaly detection
DBT_CORE	DBT_CORE_EXECUTE with dbt test;	dbt test framework

---

Airflow Operator Mapping

SqlCheckOperator → *_RUN_TEST

SqlCheckOperator passes if the SQL returns a truthy (non-zero, non-null) result; fails otherwise.

# Airflow
check_no_nulls = SqlCheckOperator(
    task_id="check_no_nulls",
    conn_id="snowflake_prod",
    sql="SELECT COUNT(*) FROM orders WHERE customer_id IS NULL",
)
# Passes if COUNT = 0, fails if COUNT > 0