using-data-engineering-agent-skills

Using Data Engineering Agent Skills

Overview

Start here before changing code, SQL, orchestration, contracts, or infrastructure. This skill maps the user request to the right data engineering workflow so the agent does not skip specification, quality, governance, replay, or rollback thinking.

When to Use

• starting a new session
• deciding which skill should lead execution
• translating a vague request into the right workflow
• choosing the best preset, starter pack, or runnable example
• deciding the safest next command

Do not stop here once the task has been classified. Load the actual execution skill after triage.

Workflow

1. Classify the task type.
   • New data product, new pipeline, or major behavior change: use data-specification
   • Approved scope that needs sequencing: use pipeline-planning-and-task-breakdown
   • Implementation across code, SQL, or jobs: use the relevant build skill plus a matching preset
   • File drops, SFTP, or partner-managed feeds: use file-and-partner-feed-ingestion
   • Glue Data Catalog, Lake Formation, or AWS-native catalog governance: use glue-data-catalog-and-lake-formation-governance
   • Python implementation work: use python-data-engineering-and-pipeline-packaging
   • Scala JVM data jobs: use scala-data-engineering-on-jvm-runtimes
   • Java connectors or data services: use java-data-engineering-and-integration-services
   • Unity Catalog or Databricks lakehouse governance: use unity-catalog-and-lakehouse-governance
   • Purview or Azure-native governance: use microsoft-purview-and-azure-data-governance
   • Dataplex, policy tags, or BigQuery governance: use dataplex-and-bigquery-governance
   • MySQL versus NoSQL or operational-store choice: use operational-datastore-selection-relational-and-nosql
   • ETL, ELT, or transformation-boundary redesign: use etl-elt-and-modernization-strategy
   • Test data generation, seeded fixtures, or lower-environment realism: use test-data-preparation-and-synthetic-data
   • Production-like data refresh into development, QA, or staging: use lower-environment-data-masking-and-obfuscation
   • Quality, reconciliation, or contract correctness: use data-quality-and-contract-testing
   • Resiliency testing, failure injection, restart drills, or failover validation: use data-resiliency-testing-and-failure-injection
   • Disaster recovery or business continuity planning: use data-platform-disaster-recovery-and-business-continuity
   • Reliability issue, incident, or broken publish: use incident-triage-and-pipeline-recovery
   • Replay, rerun, cutover, or historical repair: use safe-backfill-and-replay-orchestration, orchestration-and-backfills, and data-migration-and-platform-cutover
   • Serverless Spark on Lambda or short-lived runtimes: use spark-serverless-reliability-and-state-management
   • Kafka production guardrails, DLQs, or schema enforcement: use kafka-resilience-and-schema-evolution
   • Live lag, Spark plans, or run-state diagnosis via MCP: use mcp-data-observability-integration
   • Governance, lineage, privacy, or access changes: use lineage-pii-and-governance
   • PII, PCI, HIPAA, PHI, or audit-bound data handling: use data-security-compliance-and-regulated-data
   • region-specific compliance, sovereignty, or localization rules: use regional-data-compliance-and-sovereignty
   • sustainability or ESG reporting data products: use esg-and-sustainability-regulatory-reporting
   • Schema changes or breaking contracts: use schema-evolution-and-contract-migrations
   • Informatica, Talend, or legacy ETL modernization: use enterprise-etl-and-data-integration-modernization
   • mainframe data offload or modernization: use mainframe-modernization-and-data-offload
   • Platform-team ownership, golden paths, or support-boundary design: use data-platform-operating-model-and-service-ownership
   • quality tool selection, rule severity, or quality operating model: use data-quality-platforms-and-rule-management