What is r0b0bench AGENTS.md?

A set of instructions for r0b0bench, a command-line client that benchmarks models through an OpenAI-compatible API. It defines four benchmark profiles and their required test order.

Use it to run canary, quality, latency, concurrency, throughput, and retrieval-depth tests, while controlling chat-template settings through the runtime environment.

It prevents inconsistent benchmark runs, such as hardcoding context lengths or mixing results from different measurement methods.

How r0b0tlab/r0b0bench configures Codex and OpenCode

Reuse

Borrowing it

Nothing to install: this file belongs to r0b0tlab/r0b0bench. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/r0b0tlab/r0b0bench/main/AGENTS.md

Clone the repo

git clone --depth 1 https://github.com/r0b0tlab/r0b0bench

Made for: Codex, OpenCode.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/r0b0tlab/r0b0bench/agents-md/github.svg)](https://agentmods.dev/instructions/r0b0tlab/r0b0bench/agents-md)

Your own site

<a href="https://agentmods.dev/instructions/r0b0tlab/r0b0bench/agents-md"><img src="https://agentmods.dev/badge/instructions/r0b0tlab/r0b0bench/agents-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/r0b0tlab/r0b0bench/agents-md"><img src="https://agentmods.dev/badge/instructions/r0b0tlab/r0b0bench/agents-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 903 This file is loaded in full into every session.

When invoked 903 The same file — it is already loaded in full.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00903	$0.00903
Opus 5.5	$0.00361	$0.00361
Sonnet 5.5	$0.00181	$0.00181
Haiku 4.5	$0.00090	$0.00090

Measured 2mo ago against content hash 703db14aa8d3, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

r0b0bench AGENTS.md scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

AGENTS.md · 60 lines

How it starts

The opening of the file, as written. The whole thing — 60 lines — stays where its author put it; the contents beside it link to each section on GitHub.

AGENTS.md — working on / running r0b0bench

This file is for coding agents (Hermes, Cursor, etc.).

What this repo is

Runnable OpenAI-compatible benchmark client (r0b0bench CLI + optional container).
Not a model server. Do not bake weights into this image.
Exactly four public profiles: core | core-subset | systems | hard-subset. The first two always include the systems block; systems runs only that block; hard-subset is the hard-multiturn quality profile (MultiChallenge + BFCL-MT with canary gates at both ends, no full systems block; τ²-bench lanes removed 2026-08-22 — no value added).

Systems block (every profile)

Order: canary → bfcl_mt → bfcl_ast → latency → concurrency → throughput → niah
NIAH depths = 25% / 50% / 90% of (max_model_len − 64) from /v1/models. Do not hardcode 8k/16k/28k.
Latency/concurrency/throughput use the portable OpenAI chat backend (label method strings; never mix with vllm-bench rows).

Chat-template control (runtime variable)

Lanes never set chat_template_kwargs. The only channel is the runtime env R0B0BENCH_CHAT_TEMPLATE_KWARGS (JSON object), merged into every chat request by endpoint._chat_body and into the BFCL adapters via extra_body. With the variable unset, no template kwargs are sent and the model's served template default applies (e.g. thinking ON for thinking-default models). Record the exact value used in every run manifest; rows measured under different template states are separate ledger entries.

Hard rules

Output outside checkout — --output must not be inside the git tree (except gitignored out/). Private evidence must not be committed.
No BFCL forks — use official bfcl-eval categories/names; do not invent leaderboard-looking composites.
Canary taxonomy — stop the run on infra/auth/schema death; ordinary wrong answers are scored (canary content fails → FAIL, not silent continue for publish).
No score washing — do not weaken verifiers to make a model pass.
--skip-systems is debug-only; report must set invalid_for_publish=true.
Do not claim r0b0bench results without a package-produced report.json.
Secrets only via env; never print tokens.

Read the full file on GitHub · 60 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 60 lines · 903 tokens per session scan A 703db14aa8d3

Subscribe to this mod's changes

r0b0bench AGENTS.md is an instructions file published in the GitHub repository r0b0tlab/r0b0bench (11 stars, last pushed 21d ago), licensed MIT. It adds 903 tokens to every session, about $0.0036 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Analytics and metrics HumanEval benchmarking bfcl ifeval llm-benchmark reproducibility

r0b0bench AGENTS.md next to 6 instructions from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
r0b0bench AGENTS.mdr0b0tlab/r0b0bench	Instructions file	11	21d ago	MIT	A	903	Codex, OpenCode
vscode buildNext.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	9,719	GitHub Copilot
next.js AGENTS.mdvercel/next.js	Instructions file	143k	today	MIT	A	7,461	Codex, OpenCode
langchain AGENTS.mdlangchain-ai/langchain	Instructions file	149k	yesterday	MIT	A	4,680	Codex, OpenCode
vscode oss-third-party-notices.instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	5,516	GitHub Copilot
vscode copilot-instructions.mdmicrosoft/vscode	Instructions file	195k	yesterday	MIT	A	3,093	GitHub Copilot
deepseek-harness AGENTS.mddeepseek-ai/deepseek-harness	Instructions file	274k	2d ago	MIT	A	4,313	Codex, OpenCode

r0b0bench: Instructions file for Codex