pipeline-architect

pipeline-architect is a skill for Claude Code from mturac/hermes-supercode-skills. It costs 148 tokens per session (1,383 once invoked), scanned A, original, MIT.

A guide for designing and building data pipelines, which move information from sources into databases, warehouses, or other destinations. It covers batch jobs, real-time streams, data transformations, and database structure changes.

In plain words
What is it for?
Use it to plan or implement ETL/ELT jobs, streaming systems, schema migrations, and warehouse architectures using tools such as Kafka, Airflow, dbt, Spark, ClickHouse, BigQuery, Snowflake, or Redis Streams.
Why use it?
It helps avoid unreliable transfers, repeated records, unclear data quality, and poorly handled failures. It also helps choose an appropriate design for how data is collected and reused.

Skill for Claude Code

Written for Claude Code: shipped in a Claude Code plugin.

Part of the api-sculptor plugin — 13 skills shipped together , and of auth-architect, db-whisperer, deploy-ninja, ghost-scraper, infra-automation, mcp-conductor, obs-guardian, pipeline-architect, prediction-alpha, prompt-forge, quantum-debugger, security-sentinel

Good fit Use it to plan or implement ETL/ELT jobs, streaming systems, schema migrations, and warehouse architectures using tools such as Kafka, Airflow, dbt, Spark, ClickHouse, BigQuery, Snowflake, or Redis Streams.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/mturac/hermes-supercode-skills/pipeline-architect
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add mturac/hermes-supercode-skills --skill pipeline-architect
Clone the repo
git clone --depth 1 https://github.com/mturac/hermes-supercode-skills

Made for: Claude Code.

Or install api-sculptor, the plugin that ships this one along with the rest of its 13 skills.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for pipeline-architect

README.md
[![agentmods](https://agentmods.dev/badge/skills/mturac/hermes-supercode-skills/pipeline-architect/github.svg)](https://agentmods.dev/skills/mturac/hermes-supercode-skills/pipeline-architect)
Your own site
<a href="https://agentmods.dev/skills/mturac/hermes-supercode-skills/pipeline-architect"><img src="https://agentmods.dev/badge/skills/mturac/hermes-supercode-skills/pipeline-architect/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for pipeline-architect

Your own site · 80×15
<a href="https://agentmods.dev/skills/mturac/hermes-supercode-skills/pipeline-architect"><img src="https://agentmods.dev/badge/skills/mturac/hermes-supercode-skills/pipeline-architect.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 148 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,383 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00148 $0.01383
Opus 5 $0.00074 $0.00691
Sonnet 5 $0.00030 $0.00277
Haiku 4.5 $0.00015 $0.00138

Measured 9d ago against content hash 01a0f0072242, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-09, from the pricing page.

Security

Grade A, and why

pipeline-architect scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

skills/pipeline-architect/SKILL.md · 178 lines

How it starts

The opening of the file, as written. The whole thing — 178 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Pipeline Architect

You are a data pipeline specialist. You design and implement systems that move data reliably from source to target — whether that's batch ETL, real- time streaming, or schema migrations. Every pipeline you build is idempotent, observable, and has clear failure handling.

Design Patterns

Know these and select the right one for the use case:

Medallion Architecture — Bronze (raw) → Silver (cleaned) → Gold (business-ready). Use when building a data lakehouse or warehouse with multiple consumers who need different levels of data quality.

CDC (Change Data Capture) — Debezium, logical replication, or application-level event emission. Use when you need near-real-time sync between an OLTP database and an analytics target.

Lambda vs Kappa — Lambda uses separate batch and stream paths; Kappa uses stream-only with replayable logs. Prefer Kappa when your streaming infrastructure (Kafka) can handle reprocessing. Use Lambda when batch corrections are a hard requirement.

Idempotency — Every pipeline must produce the same result when run multiple times with the same input. This means upsert over insert, deduplication keys, and deterministic transformations.

Workflow

1. Requirements Gathering

Before designing anything, establish:

Source:

  • What format? (JSON, CSV, Avro, Protobuf, database, API)
  • What volume? (rows/sec for streaming, GB/day for batch)
  • How stable is the schema? (does it change weekly? monthly? never?)
  • What's the availability? (API rate limits, database load concerns)

Target:

  • What system? (PostgreSQL, BigQuery, ClickHouse, Snowflake, S3)
  • What query patterns will consumers use?
  • What's the retention policy?

SLAs:

  • Freshness — how recent must the data be?
  • Accuracy — what error rate is acceptable?
  • Availability — what uptime target?

2. Architecture Design

Produce a clear architecture document:

Pipeline: user_events_to_analytics
Schedule: "*/15 * * * *"  # or "streaming"

Source:
  type: kafka
  topic: user-events
  format: avro
  schema_registry: https://schema-registry:8081

Transforms:
  - name: filter_bots
    type: filter
    condition: "user_agent NOT LIKE '%bot%'"
  - name: enrich_geo
    type: lookup
    source: maxmind_db
  - name: aggregate_hourly
    type: aggregate
    group_by: [user_id, event_type]
    window: 1h

Target:
  type: clickhouse
  table: events_gold
  partition_by: toYYYYMM(event_time)
  order_by: [user_id, event_time]

Error_handling:
  dead_letter_queue: kafka://dlq-user-events
  retry_policy: 3x exponential backoff
  alert_on: error_rate > 1%

Read the full file on GitHub · 178 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 9d ago First seen · 178 lines · 148 tokens per session scan A 01a0f0072242

Subscribe to this mod's changes

pipeline-architect is a skill published in the GitHub repository mturac/hermes-supercode-skills (2 stars, last pushed 3mo ago), licensed MIT. It adds 148 tokens to every session and 1,383 once invoked, about $0.0007 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.