Professional Data Engineering Agent Skills for developing AI Agentic Data Platforms.
This repository also configures its own agents. See what de-agent-skills tells them →
Skill Claude CodeCodex
Production dbt + Trino platform — dbt-trino adapter profiles.yml (ldap/kerberos/oauth/jwt/certificate auth), all materializations (table/view/incremental/materializedview/ephemeral), incremental strategies (append/merge/delete+insert), Iceberg tableproperties in config blocks (format/partitioning/sortedby/location)…
Skill Claude CodeCodex
Optimizing dbt-generated SQL performance on Trino — ephemeral vs view vs table materialization trade-offs, CTE explosion patterns, partition-aware incremental filters (bounded watermarks), MERGE vs delete+insert strategy selection, avoiding full-refresh anti-patterns, query hints in dbt SQL (BROADCAST hint), dbt…
Skill Claude CodeCodex
Full local Trino Lakehouse Docker Compose stack — trinodb/trino coordinator+worker, Hive Metastore 3.x with PostgreSQL backend, MinIO object storage (S3-compatible), Apache Airflow, dbt-trino, Apache Superset BI, Prometheus+Grafana monitoring, healthchecks, startup ordering, persistent volumes, isolated networks…
Skill Claude CodeCodex
Trino EXPLAIN and EXPLAIN ANALYZE plan reading — DISTRIBUTED/LOGICAL/IO/VALIDATE formats, fragment types (SINGLE/HASH/ROUNDROBIN/BROADCAST/SOURCE), exchange node analysis, stage bottleneck detection, data skew identification via task row-count variance, spill detection, operator-level metrics…
Skill Claude CodeCodex
Trino federated query architecture across heterogeneous sources — cross-catalog JOIN patterns (Iceberg+PostgreSQL+MySQL+Kafka+ClickHouse), pushdown behavior per connector type, minimizing cross-catalog data movement, materializing JDBC data into Iceberg, query routing strategy, connector-specific limitations (JDBC…
Skill Claude CodeCodex
Trino data file layout optimization for Iceberg — Parquet vs ORC file format selection, target file size tuning (iceberg.target-max-file-size), row group size, Parquet/ORC column encoding choices, Bloom filter indexes, sortedby for min/max skipping, small file detection via $files metadata table, OPTIMIZE compaction…
Skill Claude CodeCodex
Production Apache Iceberg best practices with Trino — hidden partitioning (day/month/year/hour/bucket/truncate transforms), partition evolution, sorted tables, file compaction (optimize/expiresnapshots/removeorphanfiles), snapshot management, metadata tables ($snapshots/$files/$partitions/$history), schema evolution…
Skill Claude CodeCodex
Trino-based Modern Data Platform architecture design — decoupled storage/compute, Iceberg as open table format, Hive Metastore/Glue catalog, MinIO/S3 object storage, Kafka ingestion layer, dbt transformation, Airflow orchestration, Superset BI — coordinator/worker topology, catalog design, metadata-driven…
Skill Claude CodeCodex
Trino memory management and spill-to-disk tuning — query.max-memory/query.max-total-memory/query.max-memory-per-node properties, memory.heap-headroom-per-node, JVM heap sizing (80% of RAM), spill-enabled/spiller-spill-path/spill-compression-codec configuration, exchange buffer tuning, OOM kill diagnosis, memory pool…
Skill Claude CodeCodex
Trino Modern Data Stack end-to-end reference architecture — full docker-compose (Kafka + MinIO + Hive Metastore + Trino + Airflow + dbt + Superset + Prometheus + Grafana), medallion lake DDL and pipeline DAGs, Kafka→Iceberg streaming ingest, dbt project layout with Trino profiles, Superset dataset config, Prometheus…
Skill Claude CodeCodex
Trino observability and monitoring platform — JMX Prometheus exporter configuration (running queries/failed queries/OOM kills/execution latency P50/P90/P99/memory pool metrics), Grafana dashboard panels, OpenTelemetry trace propagation, query-level event listener for structured logging, Prometheus alert rules (worker…
Skill Claude CodeCodex
Trino production readiness checklist and review — coordinator HA (active/passive with load balancer), worker autoscaling, TLS/HTTPS configuration, authentication setup, resource groups for multi-tenancy, JVM sizing, spill configuration, fault-tolerant execution for batch workloads, monitoring (Prometheus/alerting)…
Skill Claude CodeCodex
Trino distributed SQL query optimization — predicate/projection/aggregation pushdown, join reordering (AUTOMATIC/ELIMINATECROSSJOINS), broadcast vs partitioned joins, dynamic filtering, CBO with ANALYZE, filter-early patterns, partition pruning, avoiding SELECT , reducing shuffle, cross-catalog query cost, session…
Skill Claude CodeCodex
Trino resource group workload governance — resource-groups.properties configuration (maxQueued/hardConcurrencyLimit/softMemoryLimit/hardCpuLimit/schedulingPolicy/schedulingWeight), hierarchical group trees, selector rules (user/source/queryType/clientTags/regex), multi-tenant tenant isolation patterns, per-user…
Skill Claude CodeCodex
Trino security hardening and data governance — TLS/HTTPS setup, internal-communication shared secret, authentication (LDAP/OAuth2/JWT), file-based access control (catalog/schema/table/column rules, column masking, row-level filtering, query kill permissions), OPA integration, Ranger for dynamic row-filter and…
Skill Claude CodeCodex
Autonomous self-healing Trino platform — Python watchdog agents for automatic worker restart detection, hung query killer, OOM-killed query resubmission, Iceberg compaction auto-trigger on small-file detection, stale statistics auto-ANALYZE, cluster memory pressure relief (kill low-priority queries), anomaly detection…
Skill Claude CodeCodex
Airbyte ELT — source/destination connectors, sync modes (Full Refresh Overwrite/Append, Incremental Append/Deduped), cursor fields, primary keys, catalog/streams, deployment (abctl, Kubernetes Helm, Airbyte Cloud), Connector Builder, Python CDK (HttpStream, IncrementalMixin), normalization (dbt-based, airbyteraw…
Skill Claude CodeCodex
Use when building, reviewing, debugging, or scaling Apache Airflow DAGs declaratively with dag-factory YAML configs — including project setup, loader configuration, defaults hierarchy, custom/provider operators, dynamic task mapping, datasets/assets, callbacks, TaskFlow decorators, Jinja2 templating, environment…
Skill Claude CodeCodex
Apache Airflow DAG authoring — DAG definition, TaskFlow API (@task/@dag decorators), operators (Bash/Python/SQL/HTTP), sensors (poke/reschedule modes), TaskGroups, dynamic task mapping (expand/partial), branching, trigger rules, XComs, Pools, callbacks, cross-DAG pipelines, best practices.
Skill Claude CodeCodex
Apache Flink — streaming architecture (JobManager/TaskManager/slots/parallelism), PyFlink Table API and SQL (Kafka/filesystem DDL, windowing TUMBLE/HOP/SESSION, aggregations), DataStream API (KeyedStream, stateful operations ValueState/MapState, process functions), event-time vs processing-time, watermark strategies…
Skill Claude CodeCodex
Apache Kafka — topics, partitions, consumer groups, producer/consumer configuration (acks, idempotence, compression, commit strategies), exactly-once semantics, Kafka Connect (source/sink connectors, SMTs, DLQ), Schema Registry, consumer lag monitoring, CLI operations, Python confluent-kafka examples, partition…
Skill Claude CodeCodex
Change Data Capture with Debezium — PostgreSQL/MySQL/Oracle CDC connectors, change event structure (before/after/op/source), snapshot modes, Kafka Connect deployment, SMT transformations, outbox pattern, Iceberg/Delta sink integration, idempotency guarantees.
Skill Claude CodeCodex
ClickHouse OLAP — MergeTree family engines (MergeTree/ReplacingMergeTree/AggregatingMergeTree/SummingMergeTree/CollapsingMergeTree), ORDER BY/PARTITION BY/PRIMARY KEY design, data skipping indexes (minmax/set/bloomfilter), TTL tiered storage, materialized views, projections, LowCardinality/Nullable/AggregateFunction…
Skill Claude CodeCodex
Dagster Software-Defined Assets (SDA) — @asset/@multiasset decorators, asset dependencies, partitions (daily/static/dynamic/multi), sensors, IO managers, declarative automation (AutomationCondition), jobs, schedules, resource definitions, deployment.
At most 3 mods per repository are shown here, and a mod shipped inside a plugin is left to that plugin's page — the rest are on their repository pages: