Professional Data Engineering Agent Skills for developing AI Agentic Data Platforms.
This repository also configures its own agents. See what de-agent-skills tells them →
Skill Claude CodeCodex
AIOps autonomous incident response agent — LLM-driven diagnosis loop (Claude tool-use agent with kubectl/SQL/Prometheus tools), alert-to-action pipeline (PagerDuty webhook → agent trigger), automated RCA generation (failure taxonomy + log correlation), self-healing action executor with approval gate, incident severity…
Skill Claude CodeCodex
AIOps capacity planning agent — predictive resource forecasting (Prophet/linear regression on Prometheus metrics), Kubernetes HPA/VPA right-sizing recommendations, node capacity headroom calculator, Kafka partition and broker capacity model, Spark executor sizing from job history, data lake storage growth forecast…
Skill Claude CodeCodex
AIOps infrastructure anomaly detection — statistical baseline (z-score/IQR/seasonal decomposition), Prometheus anomaly rules (predictlinear/stddevovertime/MAD), ML-based detection (Isolation Forest/Prophet/LSTM autoencoder on metrics), metric correlation clustering, automated alert threshold tuning from historical…
Skill Claude CodeCodex
AIOps observability copilot — natural language to PromQL/LogQL translation, alert explanation in plain English (what fired/why/impact), Grafana dashboard auto-generation from service topology, anomaly narrative generation from metric patterns, on-call context enrichment (related alerts/recent deploys/runbook links)…
Skill Claude CodeCodex
AIOps continuous platform optimization agent — autonomous optimization loop (observe→analyze→recommend→apply), Kubernetes resource rightsizing (VPA recommendation reader + apply), Kafka consumer lag auto-scale, Spark job configuration tuning from history, idle resource detection (unused deployments/PVCs/topics)…
Skill Claude CodeCodex
AIOps autonomous query cost analyzer — Trino system.runtime.queryhistory cost SQL (CPU time/peak memory/bytes scanned), ClickHouse system.querylog top expensive queries, Spark UI cost attribution by user/job/DAG, BigQuery INFORMATIONSCHEMA.JOBS cost analysis, LLM-driven query rewrite recommendations (missing partition…
Skill Claude CodeCodex
Airflow cost optimization — KubernetesPodOperator right-sizing (request vs actual CPU/memory), spot node tolerations for batch tasks, task consolidation (reduce pod-per-task overhead), pool-based concurrency control, idle worker cleanup, log retention policy (S3 lifecycle), metadata DB right-sizing, CeleryExecutor…
Skill Claude CodeCodex
Airflow high availability review — multi-scheduler setup (HA schedulers with PostgreSQL row-level locking), CeleryExecutor worker pool sizing, KubernetesExecutor pod limits, metadata DB HA (PostgreSQL HA with PgBouncer connection pooling), DAG sync strategy (git-sync sidecar/S3/GitDagBundle), log storage (S3/GCS)…
Skill Claude CodeCodex
Airflow observability — StatsD/OpenTelemetry metrics (schedulerheartbeat/taskduration/poolopenslots/dagprocessingtotalparsetime), Prometheus scraping, Grafana dashboards (DAG success rate/task duration/slot utilization), structured task logging (JSON formatter), OpenTelemetry traces for task spans, DAG SLA miss…
Skill Claude CodeCodex
Airflow production readiness — idempotent tasks (UPSERT/partition overwrite), no top-level code at DAG parse time (Variable.get inside tasks), KubernetesExecutor configuration, connection/variable management (no hardcoded secrets), SLA callbacks, retry with exponential backoff, pool management, maxactiveruns…
Skill Claude CodeCodex
Blue/green deployment for data platforms — Kubernetes blue/green with service selector swap, Argo Rollouts blue-green strategy, database schema migration compatibility (backward-compatible DDL), dbt blue/green schema swap (swapschema macro), Airflow DAG versioning (dagid with version suffix), traffic cutover and…
Skill Claude CodeCodex
DataOps CI/CD pipeline review — pipeline stages for data projects (lint/test/build/scan/deploy), trunk-based vs GitFlow branching, environment promotion (dev→staging→prod), artifact versioning (Docker image tags/dbt manifest), deployment gating (DQ checks/smoke tests/manual approval), rollback strategy, pipeline…
Skill Claude CodeCodex
Disaster recovery review for data platforms — RTO/RPO definitions, DR strategy selection (cold/warm/hot standby), Airflow metadata DB backup and restore (pgdump/WAL), Kafka topic replication (MirrorMaker2), data lake replication (S3 CRR/GCS Transfer), Kubernetes cluster backup (Velero), runbook for DR failover, DR…
Skill Claude CodeCodex
GitHub Actions optimization for DataOps — path filters to skip unchanged components, concurrency groups (cancel-in-progress), job parallelization, caching strategies (pip/npm/Maven/Docker layers), self-hosted runners for heavy workloads, matrix strategy for multi-environment tests, reusable workflows to reduce…
Skill Claude CodeCodex
Jenkins modernization for DataOps — migration from Freestyle to Declarative Pipeline, Jenkinsfile best practices, shared libraries (vars/src structure), parallel stages, Kubernetes agent pods (docker-in-docker vs kaniko), credential management (withCredentials/Jenkins Credential Store), pipeline-as-code with…
Skill Claude CodeCodex
DataOps blameless postmortem generation — severity matrix (SEV1-4), postmortem Markdown template with timeline/impact/root cause/action items, 3 filled-in examples (Kafka rebalance/dbt incremental data loss/ClickHouse schema mutation), CAPA framework (Corrective/Action/Preventive/Action), impact quantification SQL…
Skill Claude CodeCodex
Release readiness review for data platforms — pre-release checklist (schema migrations tested, backward compatibility, rollback plan), runbook completeness, monitoring and alerting configured, on-call handoff, load/soak testing results, data reconciliation between versions, feature flag strategy, go/no-go criteria…
Skill Claude CodeCodex
Root cause analysis for DataOps incidents — 5-Why analysis for pipeline failures, failure taxonomy (infrastructure/data/logic/dependency/config/concurrency), Airflow diagnosis (task state SQL/scheduler heartbeat/DagBag errors), Spark diagnosis (OOM/skew/FetchFailed/serialization), Kafka consumer lag spike RCA, data…
Skill Claude CodeCodex
Self-healing data platform — auto-restart failed Airflow DAGs (idempotency required), Kafka consumer group auto-resume after partition rebalance, circuit breaker for flapping pipelines, automatic partition backfill on gap detection, auto-scale Kubernetes resources on OOM detection, stale statistics auto-ANALYZE, DQ…
Skill Claude CodeCodex
Data platform SLA monitoring — freshness SLAs (data available by X:XX), completeness SLAs (row count variance < N%), latency SLAs (pipeline completes within N minutes), SLA breach detection (Prometheus/Soda/SQL), SLA burn rate alerting, SLA reporting dashboards, SLA miss root cause linking, cascading SLA dependencies…
Skill Claude CodeCodex
Workflow orchestration comparison and review — Airflow vs Prefect vs Dagster vs Temporal (architecture, execution model, dependency management, observability), when to choose each, migration patterns from Airflow to Prefect/Dagster, orchestrator evaluation scorecard, event-driven vs schedule-based triggering…
Skill Claude CodeCodex
Alert fatigue reduction — SLO-based alerting (burn rate vs threshold), multi-window burn rate alerts (short/long window), AlertManager inhibition and silencing, alert deduplication (groupby), routing by severity (PagerDuty for critical / Slack for warning), alert ownership labels, runbook links in annotations, alert…
Skill Claude CodeCodex
AWS data platform review — S3 data lake (lifecycle/replication/encryption/access points), EMR vs Glue ETL trade-offs, MSK Kafka configuration, RDS/Aurora for metadata, Redshift vs Athena for analytics, MWAA (managed Airflow), EKS for containerized pipelines, IAM roles (IRSA for EKS/EMREC2 instance profiles), Lake…
Skill Claude CodeCodex
Azure data platform review — ADLS Gen2 (hierarchical namespace/RBAC/lifecycle), Azure Synapse vs Databricks vs HDInsight trade-offs, Event Hubs (Kafka protocol compatible), ADF vs Azure Databricks for ETL, Azure Managed Airflow, AKS with Workload Identity (AAD Pod Identity), Key Vault for secrets, Azure Monitor/Log…
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: