Professional Data Engineering Agent Skills for developing AI Agentic Data Platforms.
This repository also configures its own agents. See what de-agent-skills tells them →
Skill Claude CodeCodex
StarRocks CDC pipeline — Flink CDC (flink-cdc-connectors) to StarRocks Primary Key table, Debezium + Kafka → Routine Load upsert, StarRocks Flink connector (exactly-once, DELETE semantics), schema change propagation, Debezium envelope parsing, multi-table CDC fan-out, dead letter queue for failed CDC events, lag…
Skill Claude CodeCodex
StarRocks high-concurrency optimization — pipelinedop tuning for BI workloads, resource groups for workload isolation (shortquery type), query queue configuration, FE connection pool, tablet scan parallelism, point query optimization (Primary Key table direct lookup), query cache for repeated dashboard queries, BE…
Skill Claude CodeCodex
StarRocks data modeling — star schema vs wide table design, fact/dimension DDL patterns, denormalization strategy for OLAP, bitmap indexes, bloom filter indexes, sort key optimization, schema review checklist, SCD type 2 with Primary Key table, aggregation table design, BI acceleration patterns, avoiding…
Skill Claude CodeCodex
StarRocks data quality — freshness queries (MAX(updatedat) vs expected SLA), duplicate detection on Primary Key tables, null anomaly SQL (column-level null rate vs baseline), volume drift detection (day-over-day ratio), cross-table referential integrity checks, data completeness for partitions, StarRocks-specific…
Skill Claude CodeCodex
StarRocks table types — Duplicate Key (append-only log/event), Aggregate Key (pre-aggregation metrics), Unique Key (upsert dedup), Primary Key (full DML upsert/delete, MVCC), CREATE TABLE DDL, key column selection, aggregate functions (SUM/MAX/MIN/REPLACE/HLLUNION/BITMAPUNION), when to use each type, storage engine…
Skill Claude CodeCodex
StarRocks EXPLAIN plan reading — EXPLAIN/EXPLAIN VERBOSE/EXPLAIN COSTS output structure, operator nodes (OlapScanNode/HashJoinNode/AggregationNode/ExchangeNode/SortNode/ProjectNode), reading fragment instances, identifying bottlenecks (shuffle exchange cost, large scan rows, memory spill), runtime filter nodes…
Skill Claude CodeCodex
StarRocks file ingestion — FILES() table function (SELECT/INSERT from S3/HDFS Parquet/ORC/CSV without CREATE TABLE), Iceberg external catalog (HMS/Glue/REST), CREATE EXTERNAL CATALOG, cross-catalog INSERT INTO SELECT, schema auto-detection, partition filter pushdown on external tables, SHOW CREATE CATALOG, external…
Skill Claude CodeCodex
StarRocks join optimization — broadcast join (small dimension to all BEs), shuffle/hash join (repartition both sides), colocate join (no network transfer, same distribution key), bucket shuffle join, nested loop join (cross join/non-equi), join strategy selection rules, runtime filter effect on joins, join hints…
Skill Claude CodeCodex
StarRocks lakehouse integration — Iceberg/Hive/Delta external catalogs (HMS/Glue/REST), cross-catalog INSERT INTO SELECT, partition filter pushdown verification, external table statistics (ANALYZE on Iceberg), writing back to Iceberg from StarRocks (3.1+), Delta Lake catalog setup, Unity Catalog (3.2+), query…
Skill Claude CodeCodex
StarRocks materialized views — synchronous MV (rollup/aggregate acceleration), asynchronous MV (CREATE MATERIALIZED VIEW REFRESH ASYNC/MANUAL/SCHEDULED, partition-aware refresh, query rewrite, nested MVs, MV on Iceberg/Hive external tables), SHOW MATERIALIZED VIEWS, ALTER/REFRESH/DROP MV, transparent query rewrite via…
Skill Claude CodeCodex
StarRocks Medallion (Bronze/Silver/Gold) architecture — Bronze raw ingestion (Duplicate Key), Silver cleansed layer (Primary Key upsert/dedup), Gold aggregated layer (Aggregate Key or MV), incremental transform patterns (INSERT OVERWRITE partition, watermark-based), layer DDL patterns, CDC Bronze landing…
Skill Claude CodeCodex
StarRocks memory tuning — BE memory architecture (memlimit, query pool, load pool, compaction pool, metadata cache), query memory limits (querymemlimit session var), memory spill to disk (spillmode), OOM analysis (be.out log), memtracker hierarchy, FE heap tuning (JAVAOPTS), Primary Key table persistent index memory…
Skill Claude CodeCodex
StarRocks partitioning strategy — RANGE partitioning (manual/dynamic/expression), LIST partitioning, dynamic partition properties (enable/timeunit/start/end/prefix/replicationnum), partition pruning verification via EXPLAIN, hot/cold tiering with storagemedium, partition lifecycle automation, SHOW PARTITIONS…
Skill Claude CodeCodex
StarRocks query optimization — vectorized execution engine, pipeline execution model, runtime filters (bloom/in/min-max), predicate pushdown, partition pruning, sort merge optimization, query hints (SETVAR/JOIN hints), session variables for optimizer control, CBO statistics (ANALYZE TABLE/AUTO ANALYZE), query cache…
Skill Claude CodeCodex
StarRocks real-time analytics — Kafka → Routine Load → Primary Key table for sub-second freshness, low-latency BI query patterns, real-time dashboard design (materialized view for pre-aggregation), metric store patterns, window-based freshness for streaming dashboards, colocate join for real-time multi-table queries…
Skill Claude CodeCodex
StarRocks real-time data modeling — Primary Key table streaming upserts (Flink/Kafka Routine Load), mutable dimension tables (partial column UPDATE), real-time pre-aggregation with Aggregate Key + streaming inserts, Routine Load for Kafka CDC, real-time materialized view refresh, low-latency BI patterns, latency vs…
Skill Claude CodeCodex
StarRocks Routine Load for Kafka — CREATE ROUTINE LOAD DDL (all PROPERTIES/KAFKA clause parameters), JSON/CSV/Avro format config, desiredconcurrentnumber tuning, SHOW ROUTINE LOAD columns, PAUSE/RESUME/ALTER/STOP, consumer lag monitoring, error log analysis, exactly-once semantics, Schema Registry for Avro, Kafka…
Skill Claude CodeCodex
StarRocks schema evolution — ALTER TABLE ADD/DROP/MODIFY/REORDER COLUMN, column type widening rules, adding columns to Aggregate/Unique/Primary Key tables, schema change types (direct vs linked vs sorted), SHOW ALTER TABLE COLUMN progress, rollback safety, backward-compatible patterns, dbt model evolution…
Skill Claude CodeCodex
StarRocks self-healing automation — auto-restart failed Routine Load jobs (detect PAUSED/CANCELLED + RESUME), rebalance tablet distribution after BE add/remove (REBALANCE command), trigger manual compaction for high-score tablets, auto-ANALYZE stale statistics, partition management (auto-drop old partitions), dead…
Skill Claude CodeCodex
StarRocks Stream Load — HTTP PUT API (curl/Python requests), CSV/JSON format parameters (columnseparator/jsonpaths/stripouterarray), columns mapping, partialupdate for Primary Key tables, label idempotency, maxfilterratio, timeout tuning, merge commit mode (3.4+), response JSON parsing, Python bulk loader pattern…
Skill Claude CodeCodex
Trino cluster health monitoring and administration — coordinator/worker health checks, REST API status endpoints (/v1/info /v1/node /v1/query /v1/cluster), JMX MBean metrics (running queries/failed queries/OOM kills/memory pool), Prometheus JMX exporter config, Grafana dashboards, query queue depth monitoring, worker…
Skill Claude CodeCodex
Airflow-orchestrated Iceberg lakehouse ETL pipelines — Bronze/Silver/Gold layer DAGs, Iceberg maintenance jobs (optimize/expiresnapshots/removeorphanfiles) as Airflow tasks, snapshot expiration scheduling, compaction DAG patterns, partition-by-partition backfill with dynamic task mapping, late-arriving data handling…
Skill Claude CodeCodex
Airflow orchestration for Trino pipelines — TrinoOperator and TrinoHook usage, trinoconnid connection setup, idempotent DAG patterns (INSERT INTO / MERGE with deterministic runids), partition-aware scheduling with Airflow logicaldate, sensor patterns (TrinoCheckOperator), SLA monitoring, retry configuration for Trino…
Skill Claude CodeCodex
Trino warehouse cost optimization — query scan cost analysis (system.runtime.queries/system.runtime.tasks), identifying expensive queries by CPU time and data scanned, scan reduction via partition pruning and file compaction, worker autoscaling patterns (scale-to-zero for batch), spot instance strategies for workers…
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: