benchmark

benchmark is a skill for Claude Code from fynnfluegge/agtx. It costs 27 tokens per session (4,053 once invoked), scanned C, original, Apache-2.0.

A guide for running SWE-bench Lite, a test set of real software-engineering tasks, against agtx coding-agent workflows.

In plain words
What is it for?
Use it to prepare and run SWE-bench Lite benchmarks and review their results. It can guide sandbox setup, tool-image preparation, binary builds, and optional token or cost tracking.
Why use it?
It organizes setup, configuration, execution, evaluation, and reporting so you can compare how well agent workflows solve standard coding tasks. It also explains required tools such as Docker, uv, tmux, and a supported coding agent.

Skill for Claude Code

Written for Claude Code: disable-model-invocation in frontmatter. Also seen: reads .claude/ paths; positional $N argument; mentions Claude Code.

Not installable: its command points at a path on the author’s own machine, so it runs nowhere else. The line is /home/bench/.local/bin.

Part of the agtx plugin — 7 skills, 1 command, 1 MCP server shipped together

Good fit Use it to prepare and run SWE-bench Lite benchmarks and review their results. It can guide sandbox setup, tool-image preparation, binary builds, and optional token or cost tracking.

Compare 6 skills from other repositories ↓
About the project

agtx is a terminal-based shared workspace where multiple coding agents coordinate through a task board. Developers use it to delegate work, run agents in parallel across git worktrees, and move tasks through different workflow phases.

fynnfluegge/agtx · 1,500 stars · on GitHub

Install

Getting it into your agent

This one installs as part of its plugin. Adding the marketplace and installing the plugin brings it with everything else the plugin ships.

Claude Code
/plugin marketplace add fynnfluegge/agtx
Claude Code
/plugin install agtx

Made for: Claude Code.

Or install agtx, the plugin that ships this one along with the rest of its 7 skills, 1 command, 1 MCP server.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for benchmark

README.md
[![agentmods](https://agentmods.dev/badge/skills/fynnfluegge/agtx/benchmark/github.svg)](https://agentmods.dev/skills/fynnfluegge/agtx/benchmark)
Your own site
<a href="https://agentmods.dev/skills/fynnfluegge/agtx/benchmark"><img src="https://agentmods.dev/badge/skills/fynnfluegge/agtx/benchmark/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for benchmark

Your own site · 80×15
<a href="https://agentmods.dev/skills/fynnfluegge/agtx/benchmark"><img src="https://agentmods.dev/badge/skills/fynnfluegge/agtx/benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 27 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 4,053 The whole file, excluding the scripts and references it only reads on demand.
Security scan C 2 findings. A grade says what 26 rules found in the file — not that it is safe.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00027 $0.04053
Opus 5 $0.00014 $0.02027
Sonnet 5 $0.00005 $0.00811
Haiku 4.5 $0.00003 $0.00405

Measured 9d ago against content hash 0fad64ea9ef9, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-08, from the pricing page.

Security

Grade C, and why

benchmark scanned grade C with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Downloads and executes remote codehighSupply chain

curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.

| **uv** | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

| **uv** | `curl -LsSf https://astral.sh/uv/install.sh \| sh` |
benchmark/skills/benchmark/SKILL.md · 387 lines

How it starts

The opening of the file, as written. The whole thing — 387 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Benchmark — SWE-bench Lite

You are a benchmark guide. Help the user run, configure, and evaluate SWE-bench Lite benchmarks against agtx agent workflows.

Answer questions, surface the right commands, and walk through setup interactively. All commands assume the user is in the benchmark/ directory.


Prerequisites

Tool Install
Docker Required for sandbox mode. macOS: Docker Desktop. Ubuntu: apt install docker.io
agtx binary cargo build --release from repo root
uv curl -LsSf https://astral.sh/uv/install.sh | sh
tmux macOS: brew install tmux. Ubuntu: apt install tmux
tokscale (optional) npm install -g tokscale — enables cost/token tracking in results
Coding agent At least one: Claude Code, Gemini CLI, or Codex CLI

One-Time Setup

cd benchmark/swebench

# Initialize Python environment (once, or after pyproject.toml changes)
uv sync

# [Sandbox only] Build the tools image (tmux + Node.js + Claude Code)
python prebake_images.py --verbose

# [Sandbox only] Build the Linux agtx binary (Ubuntu 22.04 / glibc 2.35)
bash build_linux_binary.sh

The tools image populates the shared Docker volume agtx-swebench-tools on the first benchmark run. To force a refresh after updating Claude Code:

docker volume rm agtx-swebench-tools
python prebake_images.py --force --verbose

Configuration

Config files live in swebench/configs/. Each is a standard agtx ProjectConfig TOML written to .agtx/config.toml in every cloned repo.

Minimal (no workflow):

default_agent = "claude"
workflow_plugin = "void"

Standard agtx workflow:

default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"

Sandbox-optimised (agent works directly in /testbed, no worktree):

default_agent = "claude"
workflow_plugin = "agtx"
worktree_dir = ".agtx/worktrees"
skip_worktree = true

Mixed agents (different agent per phase):

default_agent = "claude"
workflow_plugin = "agtx"

[agents]
planning = "gemini"
running  = "claude"
review   = "codex"

Read the full file on GitHub · 387 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 9d ago First seen · 387 lines · 27 tokens per session scan C 0fad64ea9ef9

Subscribe to this mod's changes

benchmark is a skill published in the GitHub repository fynnfluegge/agtx (1,500 stars, last pushed yesterday), licensed Apache-2.0. It adds 27 tokens to every session and 4,053 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it C with 2 findings (downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.