What is bundle-bench?

A benchmarking tool for comparing combinations of o9k companion tools on the same fixed set of tasks.

It is for testing companion bundles, validating an existing setup with measurements, and contributing benchmark results to the o9k project.

It replaces guesses about which tool combination works best with results from controlled tests in isolated environments.

bundle-bench — Skill for Claude Code and Codex by Bumblebiber: A benchmarking tool for comparing combinations of o9k companion…

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add skills/bumblebiber/its-over-9k/bundle-bench

Any agent

npx skills add Bumblebiber/its-over-9k --skill bundle-bench

Clone the repo

git clone --depth 1 https://github.com/Bumblebiber/its-over-9k

Made for: Claude Code, Codex.

Or install o9k-recon, the plugin that ships this one along with the rest of its 3 skills.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/bumblebiber/its-over-9k/bundle-bench.svg)](https://agentmods.dev/skills/bumblebiber/its-over-9k/bundle-bench)

Your own site

<a href="https://agentmods.dev/skills/bumblebiber/its-over-9k/bundle-bench"><img src="https://agentmods.dev/badge/skills/bumblebiber/its-over-9k/bundle-bench.svg" alt="Measured on agentmods" height="20"></a>

Per session 70 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 982 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.00070	$0.00982
Opus 5	$0.00035	$0.00491
Sonnet 5	$0.00014	$0.00196
Haiku 4.5	$0.00007	$0.00098

Measured 3d ago against content hash 993d7ec35405, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

bundle-bench scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugins/o9k-recon/skills/bundle-bench/SKILL.md · 86 lines

How it starts

The opening of the file, as written. The whole thing — 86 lines — stays where its author put it; the contents beside it link to each section on GitHub.

bundle-bench — measured bundles, not believed ones

The framework-scout trial (Step 4) answers "is this ONE candidate clean and cheap in isolation?". This skill answers the next question: which COMBINATION of companions actually performs best? Same philosophy as karpathy/autoresearch: mutate the setup, run a fixed workload, keep or discard by the numbers, repeat.

This is expensive. One combo = 5 non-interactive Claude sessions (benchmarks/run-bench.sh); a full ablation over 8 companions ≈ 15 combos. Never run it casually, never run it on the live config. Get explicit user sign-off on the combo list (= the budget) before the first run.

Ground rules (inherited from the trial protocol)

The live ~/.claude is never the test bench. Every run uses a sandboxed CLAUDE_CONFIG_DIR. The runner refuses ~/.claude outright.
Numbers beat claims. A combo "feels better" is not a result; the result JSON is.
Comparable or worthless. Only runs with identical tasks_hash, target_ref and model compare. The runner stamps all three.

Step 1 — Sandbox per combo

export SB=/tmp/o9k-bench-sandbox-<combo>
mkdir -p "$SB"
cp ~/.claude/.credentials.json "$SB"/ 2>/dev/null || true   # auth only, nothing else

Then install exactly the combo under test into the sandbox — plugins via CLAUDE_CONFIG_DIR=$SB claude plugin install …, MCP servers via CLAUDE_CONFIG_DIR=$SB claude mcp add … --scope user. Record versions; the PR needs them. The bare baseline gets credentials and nothing else.

Step 2 — Ablation plan (not brute force)

2^n combos is waste. Run in this order, stop early when the picture is clear:

bare — no plugins, no MCP. The floor.
o9k-pillars — the 6 pillars alone. Measures o9k's own overhead/payoff.
o9k+<companion> — pillars + ONE companion each (serena, hmem, superpowers, context7, ponytail, …). Measures each companion's marginal value.
Greedy: take the best single additions, combine them, measure again. A companion that helped alone but not in combination is redundancy — that's a finding, record it.

Read the full file on GitHub · 86 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

3d ago First seen · 86 lines · 70 tokens per session scan A 993d7ec35405

Subscribe to this mod's changes

bundle-bench is a skill published in the GitHub repository Bumblebiber/its-over-9k (0 stars, last pushed 7d ago), licensed MIT. It adds 70 tokens to every session and 982 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Analytics and metrics Building agents

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes.

obra/superpowers · 21 tokens

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, or modifying behavior. Explores user intent, requirements and design before implementation.

obra/superpowers · 37 tokens

auto-perf-optimize

Run agent-driven VS Code performance or memory investigations. Use when asked to launch Code OSS, automate a VS Code scenario, run the Chat memory smoke runner, capture renderer heap snapshots, take workflow screenshots, compare run summaries, or drive a repeatable scenario before heap-snapshot analysis.

microsoft/vscode · 62 tokens

chat-perf

Run chat perf benchmarks and memory leak checks against the local dev build or any published VS Code version. Use when investigating chat rendering regressions, validating perf-sensitive changes to chat UI, or checking for memory leaks in the chat response pipeline.

microsoft/vscode · 51 tokens

chat-pet-sprite-creation

Use when creating or changing VS Code chat pet sprite art, sprite sheets, state animations, eye treatments, Stable/Insiders variants, or pet transitions under src/vs/workbench/contrib/chat/browser/widget/media/chatPet.

microsoft/vscode · 53 tokens

cpu-profile-analysis

Analyze V8/Chrome CPU profiles (.cpuprofile) and DevTools trace files (Trace-.json). Use when: profiling performance, investigating slow functions, comparing code paths, finding bottlenecks, analyzing timeToRequest, understanding call trees from sampling profiler data, analyzing layout/paint/rendering, investigating…

microsoft/vscode · 71 tokens