Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
This repository also configures its own agents. See what data-engineering-agent-skills tells them →
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through GCP-native data governance workflows with Dataplex and BigQuery. Use when designing lakes, zones, policy tags, metadata quality, lineage, discovery, and governed publishing across Cloud Storage, BigQuery, Dataflow, Dataproc, and Google Cloud analytics platforms.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through analytics engineering workflows with dbt. Use when building or modifying staging models, marts, tests, snapshots, documentation, exposures, or semantic-layer-facing models.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Debezium and Kafka Connect CDC workflows. Use when streaming database changes into Kafka topics, managing connectors, snapshots, schema evolution, or downstream CDC consumers.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Delta Lake and medallion-style lakehouse design. Use when building or modifying bronze, silver, and gold layers, Delta Lake mutation patterns, streaming-to-batch lakehouse flows, or Databricks-centered serving architectures.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through DuckDB-based local analytics and development workflows. Use when prototyping models locally, validating transformations, reproducing data issues quickly, or building lightweight analytical tooling without a full warehouse.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through operating, hardening, and modernizing enterprise ETL and integration stacks such as Informatica, Talend, DataStage, SSIS, and Matillion. Use when legacy mappings, job orchestration, migration, or coexistence with modern lakehouse patterns must be handled safely.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through ESG, sustainability, and regulatory reporting data products. Use when building governed metrics, traceable evidence, and audit-ready data pipelines for frameworks such as CSRD/ESRS, BRSR, climate disclosures, or similar sustainability reporting obligations.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through ETL, ELT, and transformation-modernization decisions. Use when choosing execution boundaries, redesigning transformation layers, or moving from legacy ETL estates to warehouse- or lakehouse-centered ELT patterns.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through machine-learning data pipelines and feature serving workflows. Use when designing feature generation, offline and online consistency, training-serving parity, point-in-time correctness, or ML-oriented data product contracts.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through file-based and partner-feed ingestion workflows. Use when landing data from SFTP, managed file transfer, shared buckets, recurring flat files, manifests, or externally supplied feeds that need validation, replay safety, and publish discipline.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through AWS-native data catalog and lake governance workflows. Use when designing or reviewing Glue Data Catalog, Lake Formation permissions, governed sharing, metadata quality, and access boundaries for S3, Athena, Redshift, EMR, or Glue pipelines.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data-quality frameworks such as Great Expectations, Deequ, and Cuallee. Use when implementing framework-based validation suites, reusable checks, or evidence-driven data-quality enforcement.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through production data incidents. Use when a pipeline fails, publishes bad data, misses an SLA, partially loads, corrupts state, or requires rollback, replay, or stakeholder communication.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Java-based data engineering services and processors. Use when building connectors, ingestion services, stream processors, metadata services, JVM batch tools, or operational integrations in Java.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Enforces production Kafka guardrails including non-breaking schema evolution, dead-letter queues for poison messages, and acks=all producer durability. Use when designing or changing Kafka topics, producers, consumers, schema registry policies, or streaming recovery paths.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data versioning workflows using lakeFS or similar systems. Use when branching data, validating changes before publish, or controlling risky lakehouse operations with versioned data states.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through lakehouse table design and open table format decisions. Use when designing or changing Iceberg, Delta, Hudi, partitioning, schema evolution, compaction, or batch and streaming interoperability.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Applies governance, lineage, ownership, and sensitive-data controls to data changes. Use when a pipeline touches published datasets, regulated information, or shared business metrics.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through masking, obfuscating, and safely promoting production-like data into lower environments. Use when QA, development, or staging needs realistic data without exposing production-sensitive values.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through mainframe data modernization and offload workflows. Use when migrating or exposing data from COBOL, JCL, VSAM, IMS, DB2 for z/OS, or batch-oriented mainframe estates into modern lakes, warehouses, streaming systems, and governed data products.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through master data and entity resolution workflows. Use when matching identities across systems, defining canonical entities, resolving duplicates, or building golden records for shared downstream use.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents to wire Model Context Protocol servers for live data platform observability including Spark execution plans, OOM diagnosis, Kafka consumer lag, and orchestration run state. Use when agents need cluster metrics, streaming lag, or job diagnostics instead of blind code changes.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Microsoft Purview and Azure-native data governance workflows. Use when designing collections, scans, classifications, lineage, policy boundaries, and governed publishing across ADLS, Synapse, Data Factory, Azure Databricks, Fabric, and Azure analytics estates.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through converting exploratory notebooks into production-ready data jobs. Use when operationalizing notebooks from Databricks, Jupyter, or similar environments into tested, packaged, repeatable workflows.
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: