Professional Data Engineering Agent Skills for developing AI Agentic Data Platforms.
This repository also configures its own agents. See what de-agent-skills tells them →
Skill Claude CodeCodex
Data Contracts — datacontract.com YAML specification (schema, quality, SLA, ownership, servers, changelog), Data Contract CLI (init/test/lint/diff/breaking/publish/export), SodaCL-style quality checks embedded in contract, breaking change detection (column removal/type change/constraint tightening), CI/CD GitHub…
Skill Claude CodeCodex
Use when designing or implementing Data Vault 2.0 data models — Hubs, Links, Satellites, Reference Tables, Same-As Links, Multi-Active Satellites, Effectivity Satellites, Point-in-Time tables, Bridge tables, Business Vault patterns, hash key generation, DDL for all entity types, insert-only DML load patterns, pipeline…
Skill Claude CodeCodex
DataHub data catalog — metadata ingestion recipes (PostgreSQL/Hive/Spark/dbt/Airflow/Kafka/S3), Python SDK (DatahubRestEmitter, MetadataChangeProposalWrapper), column-level lineage (FineGrainedLineage), GMS/MCE/MCP architecture, Kubernetes Helm deployment, CLI operations, search/discovery REST API and GraphQL.
Skill Claude CodeCodex
Skill "dbt-core" from ivanshamaev/de-agent-skills, covering dbt core, when to use, project structure, dbtproject.yml and profiles.yml.
Skill Claude CodeCodex
Activate this skill when the task involves.
Skill Claude CodeCodex
Use when writing, configuring, or optimizing dbt projects targeting Trino or Starburst — covering profiles.yml setup, all authentication methods, materializations (table/view/incremental/materializedview/ephemeral), incremental strategies (append/merge/delete+insert), table properties (format/partitioning/sortedby)…
Skill Claude CodeCodex
DE architecture decision records (ADR) — ADR template, trade-off analysis framework, technology evaluation criteria, common DE architecture decisions (storage format, orchestrator, streaming vs batch, lakehouse vs warehouse), decision reversibility, stakeholder communication.
Skill Claude CodeCodex
DE cost optimization — query cost analysis (Trino/Spark/ClickHouse/BigQuery), compute right-sizing, storage tier recommendations, spot/preemptible instances, partition pruning, Z-order/clustering, materialized view economics, data lifecycle policies, FinOps tagging, cost attribution, budget alerts.
Skill Claude CodeCodex
DE blameless postmortem — incident timeline reconstruction, impact quantification, root cause chain (5 Whys), corrective actions (CAPA), postmortem document template, severity classification, SLA breach reporting, communication templates, lessons-learned facilitation, postmortem review checklist.
Skill Claude CodeCodex
Data engineering production readiness — idempotency patterns (INSERT ON CONFLICT, MERGE, partition overwrite), retry strategies (exponential backoff, Airflow retry config), SLA monitoring (Airflow slamisscallback, Prometheus PromQL), data freshness checks (Soda freshness, max eventtime), structured logging (JSON logs…
Skill Claude CodeCodex
DE pipeline root cause analysis — failure taxonomy, lineage tracing, upstream/downstream impact, log analysis patterns, Airflow/Spark/dbt failure diagnosis, data quality anomaly RCA, 5-Whys template, incident timeline reconstruction, runbook-style investigation checklists.
Skill Claude CodeCodex
Delta Lake — table DDL (CREATE/PARTITIONED BY/LOCATION/TBLPROPERTIES), DML (INSERT/UPDATE/DELETE/MERGE), MERGE patterns (upsert/SCD2/CDC), schema evolution (mergeSchema/overwriteSchema/ALTER TABLE), OPTIMIZE compaction, Z-ORDER BY data skipping, VACUUM, Time Travel (VERSION AS OF/TIMESTAMP AS OF), RESTORE, DESCRIBE…
Skill Claude CodeCodex
Docker for data engineering environments — multi-stage Dockerfiles for dbt/Spark/Airflow images, BuildKit layer caching (--mount=type=cache), private registries (ghcr.io/Harbor), docker buildx multi-platform builds, Docker Compose local data stacks (Spark+Airflow+Kafka+MinIO+Postgres), BuildKit secrets for private…
Skill Claude CodeCodex
DuckDB in-process OLAP analytics — reading Parquet/CSV/JSON/Iceberg/Delta from S3/local, SQL (window functions, PIVOT, ASOF join, QUALIFY), Python API (duckdb.connect, fetchdf, register, UDFs), extensions (httpfs/iceberg/delta/postgres), performance tuning, COPY TO, persistent DB.
Skill Claude CodeCodex
Feast feature store — feature repo layout, Entity/FeatureView/FeatureService definitions, offline store (BigQuery/Redshift/Snowflake/file), online store (Redis/DynamoDB/SQLite), point-in-time correct joins, feast materialize, feast apply, Python SDK getonlinefeatures/gethistoricalfeatures, push sources, streaming…
Skill Claude CodeCodex
Use when building or reviewing GitHub Actions CI/CD pipelines for data engineering — covering dbt slim CI (state:modified+, --defer, manifest.json), SQLFluff SQL linting with PR annotations, Airflow DAG integrity tests (pytest/DagBag), Great Expectations and Soda data quality gates, multi-stage Docker image builds for…
Skill Claude CodeCodex
Great Expectations (GX) — DataContext setup, Data Sources (Pandas/Spark/SQL/file), Expectation Suites, built-in expectations (null/uniqueness/range/set/regex/table-level/statistical), Validation Definitions, Checkpoints, Data Docs, Airflow integration, custom expectations, dbt integration, CI/CD usage, severity levels…
Skill Claude CodeCodex
Use when designing or implementing dimensional data models following Kimball methodology — fact tables (transaction/snapshot/accumulating), dimension tables, SCD types (1/2/3/4/6), star/snowflake schemas, surrogate keys, conformed dimensions, role-playing dimensions, junk dimensions, bridge tables, date dimensions…
Skill Claude CodeCodex
Data platform on Kubernetes — Spark-on-K8s (spark-submit, pod templates, dynamic allocation, RBAC), Airflow on K8s (Helm chart, KubernetesExecutor, KubernetesPodOperator, git-sync, DAG storage), resource management, namespaces, autoscaling.
Skill Claude CodeCodex
Mage AI data pipelines — block types (data loader/transformer/exporter/sensor/custom), pipeline YAML, hybrid SQL+Python blocks, triggers (schedule/event/API), streaming pipelines, dbt integration, Spark integration, Docker/Kubernetes deployment, ioconfig.yaml, pipeline variables, callbacks, backfills.
Skill Claude CodeCodex
MCP (Model Context Protocol) server development — FastMCP/Python SDK, tools/resources/prompts primitives, STDIO and Streamable HTTP transports, Claude Desktop/Claude Code client config, MCP Inspector testing, security best practices (input validation/OAuth2/confused deputy prevention), production Docker/Kubernetes…
Skill Claude CodeCodex
Use when designing or implementing Medallion (Bronze/Silver/Gold) data lakehouse architecture — layer definitions, DDL for each layer, DML load and update patterns, incremental pipeline design, deduplication strategies (rownumber/merge/hash/CDC/watermark), schema evolution, data quality gates, partitioning per layer…
Skill Claude CodeCodex
MLflow for data engineering — tracking server setup (PostgreSQL backend + S3 artifacts), experiment tracking (params/metrics/artifacts/autolog), ETL job metadata logging (row counts, DQ metrics, lineage tags), Model Registry (register/alias/promote), model serving (pyfunc/REST API/batch scoring), MLproject files…
Skill Claude CodeCodex
OpenLineage data lineage tracking — RunEvent/Job/Dataset/facet spec, Marquez backend setup, Airflow/Spark/dbt integrations, column-level lineage, custom emitters, lineage-based impact analysis.
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: