data-pipeline-architect

data-pipeline-architect is a skill for Claude Code from organvm-iv-taxis/a-i--skills. It costs 37 tokens per session (1,330 once invoked), scanned A, original, Apache-2.0.

A guide for designing data pipelines, which are workflows that move data from sources to destinations. It covers batch processing, real-time streams, scheduling, errors, scale, and data-quality checks.

In plain words
What is it for?
Use it to gather pipeline requirements, choose an architecture, plan ETL or ELT workflows, set freshness targets, select orchestration tools, and design validation and monitoring.
Why use it?
It helps plan how data should be collected, transformed, delivered, monitored, and recovered when something goes wrong. It also clarifies the trade-offs between processing data periodically and continuously.

Skill for Claude Code

Written for Claude Code: shipped in a Claude Code plugin.

Part of the example-skills plugin — 47 skills, 2 commands, 1 agent shipped together

Good fit Use it to gather pipeline requirements, choose an architecture, plan ETL or ELT workflows, set freshness targets, select orchestration tools, and design validation and monitoring.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add organvm-iv-taxis/a-i--skills --skill data-pipeline-architect
Clone the repo
git clone --depth 1 https://github.com/organvm-iv-taxis/a-i--skills

Made for: Claude Code.

Or install example-skills, the plugin that ships this one along with the rest of its 47 skills, 2 commands, 1 agent.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for data-pipeline-architect

README.md
[![agentmods](https://agentmods.dev/badge/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect/github.svg)](https://agentmods.dev/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect)
Your own site
<a href="https://agentmods.dev/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect"><img src="https://agentmods.dev/badge/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for data-pipeline-architect

Your own site · 80×15
<a href="https://agentmods.dev/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect"><img src="https://agentmods.dev/badge/skills/organvm-iv-taxis/a-i--skills/data-pipeline-architect.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 37 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,330 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. Third-party audits
  • NVIDIA SkillSpector pass 7 Sept 2026
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00037 $0.01330
Opus 5 $0.00018 $0.00665
Sonnet 5 $0.00007 $0.00266
Haiku 4.5 $0.00004 $0.00133

Measured 13d ago against content hash 918593e4b876, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-12, from the pricing page.

Security

Grade A, and why

data-pipeline-architect scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 13d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

distributions/claude/skills/data-pipeline-architect/SKILL.md · 175 lines

How it starts

The opening of the file, as written. The whole thing — 175 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Data Pipeline Architect

This skill provides guidance for designing robust, scalable data pipelines that move data reliably from sources to destinations.

Core Competencies

  • ETL vs ELT: Traditional Extract-Transform-Load vs modern Extract-Load-Transform patterns
  • Orchestration: Airflow, Dagster, Prefect, dbt for workflow management
  • Data Quality: Validation, monitoring, lineage tracking
  • Scalability: Batch vs streaming, partitioning, parallelization

Pipeline Design Process

1. Requirements Analysis

To begin pipeline design, gather:

  • Source systems and data formats (APIs, databases, files, streams)
  • Target destinations (data warehouse, lake, lakehouse)
  • Freshness requirements (real-time, hourly, daily)
  • Data volume and velocity estimates
  • Quality and compliance requirements

2. Architecture Selection

Batch Pipelines - For periodic bulk processing:

  • Schedule-driven (hourly, daily, weekly)
  • Higher latency tolerance
  • Simpler error recovery (re-run entire batch)
  • Tools: Airflow, dbt, Spark

Streaming Pipelines - For real-time requirements:

  • Event-driven processing
  • Sub-second to minute latency
  • Complex state management
  • Tools: Kafka, Flink, Spark Streaming

Hybrid Approaches - Lambda or Kappa architecture:

  • Batch layer for completeness
  • Speed layer for low latency
  • Serving layer for queries

3. ETL vs ELT Decision

ETL (Transform before Load):

  • When target has limited compute
  • When transformation reduces data volume significantly
  • When sensitive data must be masked before landing
  • Legacy data warehouse patterns

ELT (Transform after Load):

  • Modern cloud warehouses with cheap compute
  • When raw data preservation is needed
  • When transformations change frequently
  • dbt-style transformations in warehouse

4. Pipeline Components

Extraction Layer:

  • Full extraction vs incremental (CDC, timestamp-based)
  • API pagination and rate limiting
  • Connection pooling and retry logic
  • Schema detection and drift handling

Read the full file on GitHub · 175 lines

Files

What ships with it

4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 13d ago First seen · 175 lines · 37 tokens per session scan A 918593e4b876

Subscribe to this mod's changes

data-pipeline-architect is a skill published in the GitHub repository organvm-iv-taxis/a-i--skills (17 stars, last pushed 16d ago), licensed Apache-2.0. It adds 37 tokens to every session and 1,330 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

model-strategy

Multi-model orchestration and model-switching strategy. Score-based model selection, reasoning-effort routing, cross-agent delegation (Gemini, Codex, Ollama), advisor pairing, escalation triggers, permission matrix, and cost-efficiency optimization.

ellmos-ai/skills · 52 tokens

skypilot-multi-cloud-orchestration

Multi-cloud orchestration for ML workloads with automatic cost optimization. Use when you need to run training or batch jobs across multiple clouds, leverage spot instances with auto-recovery, or optimize GPU costs across providers.

foryourhealth111-pixel/Vibe-Skills · 51 tokens

skypilot-multi-cloud-orchestration

Multi-cloud orchestration for ML workloads with automatic cost optimization. Use when you need to run training or batch jobs across multiple clouds, leverage spot instances with auto-recovery, or optimize GPU costs across providers.

davila7/claude-code-templates · 51 tokens

LQF_Machine_Learning_Expert_Guide

LQF Machine Learning Expert Guide - Routed skill for ML/Statistical Modeling with Critical Discussion Mode. Triggers on: machine learning, modeling, prediction, training, classification, regression, clustering, deep learning, neural network, model evaluation, feature engineering, hyperparameter tuning, overfitting…

foryourhealth111-pixel/Vibe-Skills · 152 tokens

imaging-data-commons

Query and download public cancer imaging data from NCI Imaging Data Commons using idc-index. Use for accessing large-scale radiology (CT, MR, PET) and pathology datasets for AI training or research. No authentication required. Query by metadata, visualize in browser, check licenses.

foryourhealth111-pixel/Vibe-Skills · 62 tokens

similarity-search-patterns

Implement efficient similarity search with vector databases. Use when building semantic search, implementing nearest neighbor queries, or optimizing retrieval performance.

foryourhealth111-pixel/Vibe-Skills · 30 tokens