Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Protect personally identifiable information in data pipelines — classifying PII, choosing masking vs tokenization vs hashing vs encryption, dynamic data masking and column-level access control, and handling deletion/right-to-be-forgotten. Use when handling sensitive data, masking or anonymizing PII, meeting…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Plan and execute migrations of legacy ETL and data warehouses — stored procedures, SSIS/Informatica, or on-prem warehouses to modern stacks (dbt, Spark, cloud warehouses) — using strangler-fig phasing, parallel runs, and row/aggregate reconciliation. Use when migrating legacy pipelines or warehouses, rewriting stored…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Design analytics data models using dimensional modeling — star and snowflake schemas, fact and dimension tables, grain declaration, surrogate keys, and slowly changing dimensions (SCD Type 1/2/3). Use when designing a warehouse schema, building marts, choosing a table grain, tracking history, or deciding fact vs…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Reduce Google BigQuery cost and runtime — partitioning and clustering, minimizing bytes processed, avoiding SELECT and full scans, slot usage and reservations, approximate functions, and materialized views. Use when BigQuery queries are expensive or slow, bytes billed are high, a query scans full tables, or you need…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Optimize columnar Parquet storage for analytics — file and row-group sizing, compression codecs (Snappy/ZSTD), partitioning and file layout, column pruning and predicate pushdown, dictionary encoding, and fixing the small-files problem. Use when Parquet reads are slow or costly, files are too small/large, choosing…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Optimize slow or failing PySpark and Spark SQL jobs — partitioning and repartitioning, data skew, shuffles, broadcast joins, caching, Adaptive Query Execution, and avoiding driver collects and Python UDFs. Use when a Spark job is slow, spills, OOMs, has skewed tasks, runs a huge shuffle, or a stage hangs on a few…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Reduce Snowflake cost and latency — right-size and auto-suspend warehouses, use multi-cluster for concurrency, apply clustering keys, read the Query Profile, exploit result/warehouse caching, and control credit spend. Use when Snowflake queries are slow or expensive, warehouses spill or queue, credits are high, or you…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Optimize slow analytical SQL by reading query/EXPLAIN plans, cutting scanned data, fixing join strategy, and using partitioning, clustering, and indexes across Postgres, Snowflake, BigQuery, Databricks/Spark SQL, and Redshift. Use when a query is slow, times out, costs too much, scans too many rows/bytes, or spills to…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Build Prefect workflows — flows and tasks, retries and caching, parameters, concurrency limits, deployments and schedules, blocks for connections/secrets, and idempotent task design. Use when writing or debugging Prefect flows, scheduling runs, configuring retries/caching, or migrating scripts to Prefect orchestration.
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Review data engineering pull requests with a data-specific checklist — idempotency, correct grain, incremental logic, cost impact, data quality tests, PII handling, and backward compatibility — that generic code review misses. Use when reviewing a dbt/SQL/Spark/Airflow PR, a data pipeline change, or a new model, and…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Provision data infrastructure with Terraform — warehouses, buckets, IAM/roles, orchestration, and streaming resources — using modules, remote state, workspaces/environments, and safe plan/apply workflows. Use when writing Terraform for Snowflake/BigQuery/Redshift, S3/GCS, IAM, Airflow/MWAA, or Kafka, or structuring…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Add data quality tests to a dbt project — generic tests (unique, notnull, acceptedvalues, relationships), singular tests, unit tests, dbt-utils and dbt-expectations packages, and source freshness. Use when adding tests to dbt models, catching data quality regressions, validating assumptions, or setting up source…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Write data transformations and loads that produce the same result no matter how many times they run — using MERGE/upsert, deterministic partition overwrites, deduplication, and stable keys. Use when a retry could duplicate data, a job is not safe to re-run, a pipeline needs exactly-once effects, or loads must be…
Unknown-333/awesome-data-engineering-skills
Skill Claude CodeCodex
Template for authoring a new data-engineering skill. Replace this text with a third-person summary of what the skill does and when to use it, including concrete trigger keywords (tools, file types, tasks). Copy this folder to skills/ / and rename it.