Professional Data Engineering Agent Skills for developing AI Agentic Data Platforms.
This repository also configures its own agents. See what de-agent-skills tells them →
Skill Claude CodeCodex
Data platform self-service API — FastAPI REST API for platform operations (Kafka topic CRUD/Airflow DAG trigger/dbt job run/Trino query execution), OAuth2+JWT authentication with team-based RBAC, async job tracking with status polling, rate limiting per team, OpenAPI spec generation, Terraform provider for API…
Skill Claude CodeCodex
Internal Developer Platform (IDP) for data engineering — Backstage Software Catalog (catalog-info.yaml for pipelines/datasets/services), Software Templates (scaffold new DAG/dbt project/Kafka topic), TechDocs integration, golden path enforcement, Port.io alternative, self-service data pipeline provisioning, platform…
Skill Claude CodeCodex
Airflow + StarRocks backfill patterns — historical partition reprocessing, date range loops with INSERT OVERWRITE, Broker Load replay for S3 partitions, safe backfill with maxactiveruns=1, partial backfill resume from checkpoint, backfill idempotency via label strategy, clearing downstream tasks before rerun, parallel…
Skill Claude CodeCodex
Airflow CDC orchestration for StarRocks — watermark-based incremental sync, Routine Load health check DAG, Flink job submission via REST API, offset lag alerting, dead letter queue reprocessing, multi-table incremental sync with dependency graph, schema change detection and pipeline pause/resume.
Skill Claude CodeCodex
Airflow + StarRocks data quality gates — post-load row count validation, freshness checks (MAX(updatedat) vs expected), null explosion detection, duplicate key audit on Primary Key tables, volume anomaly detection (z-score vs 7-day avg), SODA-style SQL quality checks embedded in DAG, quarantine pattern for bad…
Skill Claude CodeCodex
Airflow + StarRocks ETL best practices — idempotent DAG design (label strategy/partition overwrite), retry with backoff, SLA callbacks, dynamic partition creation, duplicate prevention with MERGE/INSERT OVERWRITE, DAG-level concurrency controls, dependency ordering, catchup safety, data lineage tagging.
Skill Claude CodeCodex
Airflow + StarRocks pipeline orchestration — MySqlHook for DDL/DML, Broker Load trigger + poll pattern, Stream Load via HTTP operator, Routine Load job lifecycle (pause/resume/stop), sensor for load completion, ANALYZE TABLE post-load, partition-aware DAG templating, Airflow variables for StarRocks connection config…
Skill Claude CodeCodex
Skill "dbt-starrocks-models" from ivanshamaev/de-agent-skills, covering dbt + starrocks models, when to use, installation and setup, dbt-starrocks requires dbt-core >= 1.5.0 and profiles.yml.
Skill Claude CodeCodex
The most impactful optimization: ensure incremental models only scan new partitions.
Skill Claude CodeCodex
-- Create dbt user CREATE USER dbtprod IDENTIFIED BY 'strongpasswordhere'.
Skill Claude CodeCodex
StarRocks Tables (Physical Layer) ├── silver.orders (fact) ├── silver.dimcustomers (dimension) └── gold.ordersdaily (pre-aggregated for performance) │ dbt Semantic Layer (Logic Layer) ├── semanticmodels/ (entity/measure/dimension definitions) ├── metrics/ (business metric definitions) └── savedqueries/ (pre-defined…
Skill Claude CodeCodex
Skill "dbt-starrocks-testing" from ivanshamaev/de-agent-skills, covering dbt + starrocks testing, when to use, generic tests (schema.yml), models/silver/schema.yml and source freshness tests.
Skill Claude CodeCodex
StarRocks backup and restore — CREATE REPOSITORY (S3/HDFS/NFS), BACKUP DATABASE/TABLE, SHOW BACKUP/RESTORE status, RESTORE with partition granularity, cross-cluster restore, incremental backup strategy, snapshot lifecycle, disaster recovery RTO/RPO targets.
Skill Claude CodeCodex
StarRocks cluster health monitoring — FE/BE/CN node status, tablet health, replica status, compaction backlog, storage imbalance, SHOW commands, system tables (informationschema/sys), alert thresholds, FE quorum checks, BE disk usage, dead replica detection, Prometheus metrics.
Skill Claude CodeCodex
StarRocks compaction tuning — base compaction vs cumulative compaction, compaction score analysis, write amplification reduction, BE config parameters (compactionthreads/mincumulativecompactionnumsingletondeltas/maxcompactioncandidatenum), SHOW PROC '/compactions', manual compaction trigger, Primary Key table…
Skill Claude CodeCodex
StarRocks query monitoring and resource management — slow query log, SHOW PROCESSLIST, informationschema.queryhistory, resource groups (CREATE RESOURCE GROUP), classifiers, CPU/memory/concurrency limits, query queuing, KILL QUERY, workload isolation, Audit Loader plugin.
Skill Claude CodeCodex
StarRocks security and RBAC — CREATE USER/ROLE, GRANT/REVOKE privileges (catalog/database/table/view/MV/function/resource group level), built-in roles (clusteradmin/dbadmin/useradmin/public), LDAP/LDAP group auth, row-level security policies, column masking, SSL/TLS, audit log, privilege inheritance, security best…
Skill Claude CodeCodex
StarRocks storage balancing — tablet distribution across BEs, disk skew detection (SHOW BACKENDS disk usage), clusterbalance analysis (SHOW PROC '/clusterbalance'), balanceloadscorethreshold tuning, tablet migration monitoring, BE decommission, adding new BE nodes, storage capacity forecasting, imbalance alerts.
Skill Claude CodeCodex
StarRocks aggregation optimization — one-phase vs two-phase aggregation, pre-aggregation pushdown to scan, Aggregate Key table automatic merge, GROUP BY on distribution key (local aggregation), COUNT DISTINCT optimization (multi-phase), HLL/BITMAP approximate aggregation, aggregation spill tuning, EXPLAIN aggregation…
Skill Claude CodeCodex
StarRocks AI incident RCA — autonomous root cause analysis agent for FE/BE failures, memory spike diagnosis (BE heap/process memory), Routine Load pause RCA (ErrorLogUrls + error patterns), Kafka consumer lag spike analysis, compaction backlog detection, tablet replication failures, OOM kill patterns, query timeout…
Skill Claude CodeCodex
StarRocks AI query autotuner — autonomous SQL optimization agent workflow (EXPLAIN COSTS → analyze → recommend), materialized view recommendation from slow query log, index recommendation (bitmap/bloom filter), partition pruning diagnosis, join order hints generation, statistics staleness detection and auto-ANALYZE…
Skill Claude CodeCodex
StarRocks Broker Load — LOAD DATA from S3/HDFS/GCS/MinIO/Azure, CSV/Parquet/ORC format support, column mapping expressions, multi-table atomic load, SHOW LOAD status polling, CANCEL LOAD, S3 credential patterns (access key/IAM role/instance profile), parallelism tuning, Airflow integration, wildcard file paths.
Skill Claude CodeCodex
StarRocks bucketing (distribution) — DISTRIBUTED BY HASH vs RANDOM, bucket count selection formula, distribution key selection (high cardinality/join key), skew detection and prevention, colocate join setup (colocatewith property), bucket count modification (ALTER TABLE), auto bucket sizing (StarRocks 3.2+), tablet…
Skill Claude CodeCodex
StarRocks cost-based optimizer — ANALYZE TABLE (full/sample/histogram/predicate columns), AUTO ANALYZE configuration, statistics storage (statistics database), SHOW ANALYZE STATUS, stale statistics detection, cardinality estimation failures, CBO join hints (LEADING/broadcast/shuffle), EXPLAIN COSTS reading…
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: