What is run-benchmark?

A workflow for running benchmark tests that compare multiple prompt or agent variants against a codebase. It uses dry runs, where agents propose changes without editing files, and collects their results for comparison.

It helps run one dry-run agent per variant, enforce no-write rules, retrieve session logs, and compile benchmark metrics.

It prevents inconsistent testing and makes it easier to see which variant performs better.

How loerei/chronicle-mcp configures Claude Code and Codex

Reuse

Borrowing it

Nothing to install: this file belongs to loerei/chronicle-mcp. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/loerei/chronicle-mcp/main/.agents/skills/run-benchmark/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/loerei/chronicle-mcp

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/loerei/chronicle-mcp/run-benchmark/github.svg)](https://agentmods.dev/skills/loerei/chronicle-mcp/run-benchmark)

Your own site

<a href="https://agentmods.dev/skills/loerei/chronicle-mcp/run-benchmark"><img src="https://agentmods.dev/badge/skills/loerei/chronicle-mcp/run-benchmark/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/loerei/chronicle-mcp/run-benchmark"><img src="https://agentmods.dev/badge/skills/loerei/chronicle-mcp/run-benchmark.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 44 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 900 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00044	$0.00900
Opus 5	$0.00022	$0.00450
Sonnet 5	$0.00009	$0.00180
Haiku 4.5	$0.00004	$0.00090

Measured 8d ago against content hash 02b6e3661298, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-09, from the pricing page.

Security

Grade A, and why

run-benchmark scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.agents/skills/run-benchmark/SKILL.md · 70 lines

How it starts

The opening of the file, as written. The whole thing — 70 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Run Benchmark

Quick Start

Parse the prompts from the benchmark sheets (e.g., Calcom Prompts Benchmark.txt).
Spawn dry-run subagents for each variant.
Fetch full session details and compile comparisons.

Workflows

1. Spawning Subagents

You MUST spawn one subagent per prompt variant. The subagents MUST run in a dry-run environment.

Use invoke_subagent with:
- TypeName: "self"
- Workspace: "branch" (prevents file locks and race conditions)
- Prompt: The exact prompt text from the benchmark sheet.

2. Guarding the Dry-Run

Ensure every prompt contains:

Strict Dry-Run Prefix: [DRY RUN ONLY - NO MCP SERVERS ALLOWED - NO GLOBAL DOCS - DO NOT CALL WRITE/PATCH/EDIT TOOLS - PROJECT ROOT: <path>] [RULE: Only output the proposed diff. Do NOT write or patch any files on disk.]
If using metadata: [RULE: Trust HoverSource Metadata - Proceed directly to viewing the specified file and line without search or verification. Do NOT write or patch any files on disk. Only output the proposed diff.]

3. Collecting and Saving Logs

Once all subagents in a set complete, you MUST retrieve and save their session logs.

Save Session Log: Call chronicle-mcp:get_session_details with:
- sessionId: The subagent's conversation ID.
- includeCallResults: true
- includeToolCalls: true
- output: Absolute path to save the markdown file (e.g. D:/Projects/HoverSource/benchmark-logs/calcom-task1-a.md).

4. Compiling Benchmark Report

Generate or update the consolidated benchmark table (e.g. calcom-benchmark.md).

Table Structure Requirement: The tables MUST be formatted exactly like the project's main README.md benchmark tables:
- Rows: Metrics (Task Achievement, Agent Steps, Tool Calls, Execution Time, Cumulative Input, Peak Context).
- Columns: Prompt Variants on the left, Delta columns on the far right. Format: | Metric | Pure Natural Language (A) | Senior Developer (B) | HoverSource Metadata (C) | Delta (B vs A) | Delta (C vs A) |
- Task Achievement Metric:
  - For individual task tables: Record 1 (Succeeded - modified correct file & logic) or 0 (Failed - modified wrong file/project or wrong logic).
  - For the consolidated summary table: Calculate the average success rate percentage (e.g. 71.4% (5/7)).
- Delta Calculations:
  - Delta (%) = ((Variant - A) / A) * 100 (formatted with a bold -% or +% value).
  - For Time and Tokens, also include the multiplier (e.g. 13.6x fewer/faster).

Read the full file on GitHub · 70 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

8d ago First seen · 70 lines · 44 tokens per session scan A 02b6e3661298

Subscribe to this mod's changes

run-benchmark is a skill published in the GitHub repository loerei/chronicle-mcp (0 stars, last pushed 4d ago), licensed MIT. It adds 44 tokens to every session and 900 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Analytics and metrics Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

chronicle-mcp: Skill for Claude Code