Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
This repository also configures its own agents. See what data-engineering-agent-skills tells them →
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through metadata platform and lineage workflows using OpenMetadata, DataHub, or OpenLineage-compatible systems. Use when improving discovery, lineage quality, metadata governance, or producer-to-catalog integration.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through choosing relational operational stores such as MySQL versus NoSQL options such as document, key-value, wide-column, or cache-backed systems. Use when deciding where application-adjacent or pipeline-adjacent operational data should live.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Designs scheduling, reruns, and backfills safely for data systems. Use when changing orchestration, retries, dependency timing, historical reprocessing, or publish sequencing.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Breaks approved data specifications into safe, verifiable implementation tasks. Use when a data project spans multiple steps, systems, or files and needs dependency-aware sequencing.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through privacy, retention, and deletion workflows in data systems. Use when handling personal data, retention limits, deletion requests, legal holds, or data minimization requirements across pipelines and published datasets.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Python-based data engineering implementation. Use when building or modifying Python ingestion jobs, orchestration helpers, PySpark entry points, validation code, packaging, dependency management, or operational CLI workflows.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through region-specific data compliance, residency, sovereignty, and transfer design. Use when data products must operate across jurisdictions such as Europe, the USA, India, Saudi Arabia, or regulated sectors with local supervisory overlays like SAMA.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through reverse ETL and operational data serving workflows. Use when sending curated warehouse data to business systems, SaaS tools, APIs, activation layers, or operational applications that rely on stable downstream contracts.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Forces replay-safe rollout plans, reconciliation gates, and rollback paths before executing any data backfill or historical reprocessing. Use when running /backfill, rerunning pipelines, repairing publish windows, or any work that risks double-counting or downstream corruption.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Scala-based data engineering on JVM runtimes. Use when building Spark, Flink, Kafka Streams, or other Scala data jobs that require explicit build, packaging, type, and runtime discipline.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through schema changes and contract migrations. Use when adding, renaming, removing, or changing columns, data types, nullability, keys, or downstream-facing data contracts.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through semantic layer and shared metric design. Use when defining business metrics, reusable dimensions, governed metric contracts, or shared semantic models consumed by dashboards, analytics tools, or other teams.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Snowflake-native pipeline and governance workflows. Use when building or reviewing Snowflake pipelines with Streams, Tasks, Dynamic Tables, Snowpipe, Snowpark, masking policies, row access, secure sharing, and warehouse-native operational controls.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through source reliability and extraction resilience. Use when upstream systems are flaky, slow, rate-limited, late, or operationally unreliable and ingestion behavior must remain safe and observable.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through batch and distributed data processing design. Use when implementing or reviewing Spark-based pipelines, or managed distributed runtimes such as Glue and EMR.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Enforces timeout-aware rollbacks, resumable checkpoints, and orphan cleanup for serverless Spark workloads on AWS Lambda, Glue, and similar runtimes. Use when writing or reviewing Spark jobs in serverless environments, S3 checkpoint patterns, partial-failure recovery, or IceGuard-style state management.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through event streaming and real-time data pipeline design. Use when working with Kafka, Kinesis, Flink, stream processing, windowing, stateful consumers, or near-real-time publish flows.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Superset and metrics-serving workflows. Use when publishing governed metrics to Superset, defining semantic consistency for dashboards, or managing chart-ready analytical datasets.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through infrastructure as code for data platforms. Use when provisioning or modifying storage, roles, secrets, networking, orchestration resources, catalogs, compute, or environment-specific data platform foundations.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through preparing test data, masked fixtures, and synthetic datasets for development, QA, and lower environments. Use when teams need representative but safe data for validation, demos, training, or release testing.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Trino and Presto federated query design. Use when querying across heterogeneous systems, planning semantic consistency, or managing performance and governance in federated analytics.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Unity Catalog governance workflows for Databricks lakehouses. Use when defining catalogs, schemas, external locations, access boundaries, shares, lineage, and cross-workspace governance for governed Delta Lake, ML, analytics, and serving assets.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Provides a compatibility alias for the main data engineering entry skill. Use when an existing setup still points at the legacy start-here file.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Helps agents classify data engineering work, choose the right preset and skill bundle, and pick the safest next command. Use when starting a session, triaging an ambiguous request, or deciding how to proceed.
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: