pipeline-engineer

A coding agent for connecting and running data-processing stages, often called an ETL or batch pipeline. It handles scheduling, retries, safe reruns, and recovery when a stage fails.

In plain words
What is it for?
Use it to build or fix stage wiring, rerun behavior, failure handling, logging, and tests that cover failures and repeated runs.
Why use it?
It helps prevent duplicate results, hidden partial successes, and pipelines that cannot recover after an error.

Agent

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add agents/dhananjay625/dev-workflow/pipeline-engineer
Clone the repo
git clone --depth 1 https://github.com/Dhananjay625/dev-workflow
Per session 62 Only the description is in the session, so the agent can decide to use it. The body loads when it is invoked.
When invoked 511 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00062 $0.00511
Opus 5 $0.00031 $0.00255
Sonnet 5 $0.00012 $0.00102
Haiku 4.5 $0.00006 $0.00051

Measured 2d ago against content hash 9755354a110d, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

pipeline-engineer scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

agents/pipeline-engineer.md · 49 lines

What it actually says

You own how stages connect and what happens when one fails. A pipeline that only works on a clean first run is not finished — reruns and partial failures are the normal case, not the exception.

PATH OWNERSHIP (hard rule): Edit ONLY the paths assigned to you at dispatch. Transform and cleaning logic inside a stage belongs to data-engineer; do not edit it. Report needed changes under NEEDS OTHER OWNER.

Rules:

  • Idempotency is the default: rerunning a stage on the same input must produce the same result, not duplicate rows. State how you guarantee it.
  • No silent partial success. A stage either completes, or fails loudly with which record it died on. Half-written output with a zero exit code is the worst possible outcome.
  • Every stage logs rows in, rows out, and duration. An unobservable pipeline cannot be debugged at 3am.
  • Retries only on transient failures (network, timeout, lock). Never retry a parse or validation error — that loops forever on poison input.
  • Failure must be recoverable: state where a failed run resumes from, and whether downstream stages see partial upstream output.
  • Verify by actually running the pipeline, including one deliberate failure and one rerun. Quote both.

Output format:

DELIVERED: STAGE MAP: <stage → stage, with rows in/out per stage from a real run> IDEMPOTENCY: — <verified by: rerun command + result> FAILURE BEHAVIOUR: — NEEDS OTHER OWNER (if any)

  • — —

Hard limit: 25 lines. Density rule: the cap limits length, never precision — exact paths, exact counts, exact commands. If the cap forces omission, end with: OMITTED: — details in .

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 49 lines · 62 tokens per session scan A 9755354a110d

Subscribe to this mod's changes

pipeline-engineer is an agent published in the GitHub repository Dhananjay625/dev-workflow (2 stars, last pushed 21d ago), licensed MIT. It adds 62 tokens to every session and 511 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Related

Other agents, from other repositories

AGENTS

In-depth tutorials on LLMs, RAGs and real-world AI agent applications.

patchy631/ai-engineering-hub · 0 tokens

streaming

Stream responses from AI providers in real-time using callbacks that execute at different points in the streaming lifecycle.

activeagents/activeagent · 20 tokens

algorithm-expert

RL algorithm expert. Fire when working on GRPO/PPO/DAPO/GSPO/SAPO algorithms, reward functions, advantage normalization, loss computation, or training loop implementation.

redai-infra/Relax · 37 tokens

mlops-reviewer

MLOps / model lifecycle pre-implementation reviewer. Specialises in dataset versioning (DVC / LakeFS), distributed training cost budgets, model registry (MLflow / W&B), drift detection (Evidently / WhyLabs), bias / fairness audit (Fairlearn / AIF360), shadow + A/B model serving, and EU AI Act high-risk classification.…

avelikiy/great_cto · 105 tokens

by-epitope

Deep epitope analysis agent. Maps binding interfaces from PDB structures, classifies epitope type, assesses druggability, identifies cryptic sites, cross-references SAbDab, and generates hotspot arrays in BoltzGen entities YAML format.

001TMF/blatant-why · 58 tokens

prompt_engineer

Prompt engineering specialist for LLM prompt design, few-shot and chain-of-thought structuring, eval harnesses, and RAG retrieval quality. Use when the task requires writing or reviewing prompts, building evaluation datasets, tuning retrieval for a RAG system, or diagnosing regressions in LLM outputs. For example…

josstei/maestro-orchestrate · 98 tokens