Production-grade Agent Skills for data engineering AI agents: 73 workflows, platform presets, safe backfill/replay, Kafka & Spark reliability, MCP observability, and VS Code/JetBrains installers.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through workflow orchestration design and operation across Airflow-style DAGs, cloud-native schedulers, and event-driven pipeline control planes. Use when building or modifying workflow dependencies, retries, triggers, sensors, SLAs, or cross-system pipeline coordination.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Apache Beam pipelines that unify batch and streaming logic. Use when designing Beam transforms, windowing, runners, replay behavior, or portability across execution backends.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through Apache Hudi lakehouse design. Use when managing incremental upserts, record-level mutations, timeline behavior, compaction, and Hudi-based lakehouse tables.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through API and SaaS ingestion workflows. Use when extracting data from REST, GraphQL, or SaaS platforms with pagination, rate limits, auth rotation, backfills, or unstable source contracts.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through schema-registry-backed event contracts. Use when managing Avro, Protobuf, or JSON Schema for event streams, compatibility policies, producer and consumer evolution, or contract enforcement in messaging systems.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through BigQuery- and Dataform-centered data engineering workflows. Use when designing BigQuery physical models, ingestion boundaries, Dataform transformation workflows, slot or cost controls, and platform decisions across BigQuery, Dataflow, Dataproc, and GCP orchestration services.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through change data capture and incremental load design. Use when building or modifying watermark-based loads, upserts, deduplication, merge logic, late data handling, or replayable incremental pipelines.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through ClickHouse-based real-time analytics design. Use when building fast analytical serving layers, event aggregations, materialized views, or low-latency metric access patterns.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data catalog, discovery, and metadata quality workflows. Use when publishing datasets, improving discoverability, curating lineage metadata, or making data products easier for other teams to find and trust.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data-contract testing using schema registries and compatibility checks. Use when validating event contracts, stream schema evolution, consumer compatibility, or release gates for schema-managed systems.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data lake and zone architecture design. Use when defining raw, refined, curated, or publish layers; storage organization; retention; and operational boundaries for a data lake.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through domain-oriented data product and data mesh design. Use when organizing ownership, domain boundaries, federated governance, and shared platform responsibilities across multiple teams.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data migration and platform cutover workflows. Use when moving pipelines, tables, contracts, orchestration, or workloads between systems, clouds, warehouses, lakehouses, or serving layers.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data observability and service-level management. Use when defining or improving freshness, completeness, anomaly detection, alerting, lag tracking, run metadata, and ownership for production data products.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through CI/CD and release management for data platforms. Use when promoting pipeline code, SQL models, contracts, infra, or configuration across environments with validation gates, staged rollout, and rollback awareness.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through disaster recovery and business continuity planning for data platforms. Use when defining region or account failover, backup and restore, RTO or RPO targets, control-plane recovery, or restore drills for pipelines, warehouses, lakehouses, and publish surfaces.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data platform operating model and ownership design. Use when defining platform team responsibilities, service tiers, golden paths, escalation boundaries, onboarding flows, or handoffs between central platform teams and domain or product teams.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Drives data implementation with contracts, assertions, and validation evidence. Use when adding or changing ingestion logic, transformations, schemas, or published data products.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through data-quality operating models and tool selection. Use when designing rule portfolios, severity levels, ownership, evidence, and enforcement across dbt tests, Great Expectations, Deequ, Cuallee, Soda, warehouse-native checks, and platform monitoring workflows.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through reconciliation and control design for business-critical data. Use when validating financial, operational, or audit-sensitive metrics with source-to-target totals, control balances, exception tracking, or close-process dependencies.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through resiliency testing for data platforms. Use when designing or running failure drills, recovery validation, failover tests, replay-safety checks, dependency outage exercises, or fault injection for pipelines and publishes.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through regulated-data security and compliance workflows for PII, PCI, HIPAA, PHI, and similar obligations. Use when data products handle sensitive fields, regulated records, control evidence, or audit-bound publish paths.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Guides agents through publishing data products for internal or external consumers. Use when sharing tables, files, extracts, APIs, or reverse-ETL-ready outputs that require stable contracts, ownership, and communication.
vaquarkhan/data-engineering-agent-skills
Skill Claude CodeCodex
Creates structured specifications for data products and pipeline changes. Use when starting a new pipeline, model, ingestion flow, or any significant change with unclear requirements.
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: