airflow

Apache Airflow AI Skill Guide

Overview & Engine Architecture

Airflow schedules DAGs of tasks executed by workers; the scheduler parses DAG files, the metadata DB stores run state, and executors (Local/Celery/Kubernetes) run task instances. Agents write idempotent tasks, set explicit retries/timeouts, avoid top-level heavy I/O in DAG files, and pass data via XCom sparingly (or external storage).

DAG file -> scheduler -> executor/workers
                |
           metadata DB (runs, XCom)
                |
           task logs / sensors

When to use this skill

• Time-based or data-aware batch pipelines
• Orchestrating dbt, Spark, warehouse SQL, ML batch jobs
• Backfills with clear logical dates

Operational directives

1. Keep DAG top-level code fast (imports + structure only).
2. Tasks must be idempotent for a given data_interval / logical date.
3. Set retries, retry_delay, and execution_timeout intentionally.
4. Prefer pushing large payloads to object storage over big XComs.
5. Never commit connection passwords; use Airflow Connections / secrets backend.

Minimal DAG

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="orders_daily",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
    default_args={"retries": 2, "retry_delay": timedelta(minutes=5)},
    tags=["orders"],
) as dag:
    extract = BashOperator(
        task_id="extract",
        bash_command="python /opt/airflow/jobs/extract_orders.py --date {{ ds }}",
    )
    dbt_run = BashOperator(
        task_id="dbt_run",
        bash_command="cd /opt/dbt && dbt build --select marts.* --vars '{run_date: {{ ds }}}'",
    )
    extract >> dbt_run

Useful CLI

airflow dags list
airflow dags test orders_daily 2026-08-26
airflow tasks test orders_daily extract 2026-08-26

Common failures

Symptom	Cause	Fix
DAG not appearing	import error / parse fail	check scheduler logs
Zombie / stuck tasks	worker death	timeouts; health checks
Huge backfill load	catchup=True	limit; clear carefully
Sensor hanging	wrong poke / mode	reschedule mode; timeouts