A generator of ADE-Bench benchmark tasks from a dbt project. dbt is a tool for building and testing data models with SQL, while a benchmark task tests whether an AI agent can solve a realistic problem.

Scanning dbt models and producing task files, setup and solution scripts, patches, and SQL tests for different difficulty levels.

It turns an existing working project into repeatable tests by creating controlled bugs and checking whether the agent repairs them correctly.

create-task — Skill by typedef-ai

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add skills/typedef-ai/ade-bench-plugin/create-task

Any agent

npx skills add typedef-ai/ade-bench-plugin --skill create-task

Clone the repo

git clone --depth 1 https://github.com/typedef-ai/ade-bench-plugin

Made for: Claude Code, Codex.

Per session 66 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 7,169 The whole file, excluding the scripts and references it only reads on demand.

Security scan B 1 finding. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.00066	$0.07169
Opus 5	$0.00033	$0.03585
Sonnet 5	$0.00013	$0.01434
Haiku 4.5	$0.00007	$0.00717

Measured 2d ago against content hash 8c50118e7e72, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade B, and why

create-task scanned grade B with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Recursive force deletemediumDestructive command

rm -rf with a variable or a broad path is one typo away from removing the wrong tree.

rm -rf /tmp/ade-bench-validate-{task_id}

Downgraded: this mod is about security review, or the phrase is quoted, so it is likely naming the pattern rather than instructing it.

skills/create-task/SKILL.md · 616 lines

How it starts

The opening of the file, as written. The whole thing — 616 lines — stays where its author put it; the contents beside it link to each section on GitHub.

ADE-Bench Task Generator

You are a benchmark task generator for ADE-Bench, a framework that evaluates AI agents on dbt and SQL tasks. Your job is to analyze a user's dbt project, propose realistic bug-injection tasks, and generate the full scaffolding needed to run those tasks as benchmarks.

Overview

ADE-Bench tasks work by:

Taking a working dbt project
Introducing a bug via a patch (setup)
Giving an agent a prompt describing the symptom
Evaluating whether the agent fixes the bug (via dbt tests + table comparison)

You will generate tasks that follow this pattern using the user's own dbt project and data.

Phase 1: Parse Arguments and Discover the Project

The user invokes this skill with a path to their dbt project. Parse the arguments:

First positional arg: Path to the dbt project directory (required). This directory must contain dbt_project.yml.
--output-dir: Where to write generated tasks (default: ./ade-bench-tasks next to the project).
--no-copy: If present, reference the original project in-place instead of copying it. Default: copy into {output_dir}/projects/.
--db-type: Database type override (duckdb or snowflake). Default: auto-detected from profiles.yml.
--db-name: Name of the database (without extension for DuckDB, or the Snowflake database name). Default: inferred from profiles.yml.
--db-path: Path to the directory containing the DuckDB file (DuckDB only). When provided, this value is written to db_dir in task.yaml. When omitted, do NOT write db_dir at all — the harness falls back to ~/.ade-bench/shared/databases/duckdb/<db_name>.duckdb, which is correct for any bundled ade-bench project (airbnb, f1, quickbooks, etc.).

Verify ade-bench is Installed

Before discovery, check whether ade-bench (the harness that runs the generated tasks) is available. Generation works without it, but the user can't run the output without it.

Detection — in order:

command -v ade — if it returns a path and ade --help succeeds, ade-bench is installed.
test -f ~/.ade-bench/pyproject.toml && grep -q 'name = "ade-bench"' ~/.ade-bench/pyproject.toml — fallback if ~/.local/bin isn't on PATH yet.

Read the full file on GitHub · 616 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2d ago First seen · 616 lines · 66 tokens per session scan B 8c50118e7e72

Subscribe to this mod's changes

create-task is a skill published in the GitHub repository typedef-ai/ade-bench-plugin (3 stars, last pushed 3mo ago), licensed MIT. It adds 66 tokens to every session and 7,169 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it B with 1 finding (recursive force delete). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Analytics and metrics Debugging Testing ai-agents analytics-engineering benchmark claude-code claude-code-plugin dbt

pr-verify

Verify a Docglow change actually works before submitting or merging a PR. Runs the conformance suite, then a behavioral verification pass (flag matrix, artifact-join spot checks, pipeline contract sweep, payload budget). Use when reviewing a PR, self-reviewing a branch before opening a PR, or when asked to "verify…

docglow/docglow · 79 tokens

dbt-agent-readiness

Audit a dbt project for agent-readiness: what would an AI agent get wrong if you pointed it at this data today? Produces a prioritized report organized by failure modes (wrong numbers, wrong table, wrong column, can't join, query fails). Scales via two-pass architecture with parallel subagents. Each subagent reads its…

GetCassis/dbt-agent-readiness · 126 tokens

authoring-meta-context

Authors and extracts dbt MetricFlow meta context — structured YAML meta: blocks that encode business knowledge (thresholds, investigation paths, SLAs, relationships) alongside metric definitions so AI agents answer analytical questions accurately. Use when user asks about context cards, meta context blocks, dbt metric…

keithbinkly/dbt-meta-context · 97 tokens

create-task