What is PostTrainBench AGENTS.md?

Repository instructions for PostTrainBench, a framework that measures how well AI agents improve small language models on set benchmarks.

Use them when adding an agent, working on model post-training experiments, changing benchmark evaluation, or using the project’s job and analysis utilities.

They explain the project layout, important files, and conventions so an agent can make changes without overlooking the training, evaluation, or cluster tooling.

How aisa-group/PostTrainBench configures Codex and OpenCode

Reuse

Borrowing it

Nothing to install: this file belongs to aisa-group/PostTrainBench. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/aisa-group/PostTrainBench/main/AGENTS.md

Clone the repo

git clone --depth 1 https://github.com/aisa-group/PostTrainBench

Made for: Codex, OpenCode.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/instructions/aisa-group/posttrainbench/agents-md/github.svg)](https://agentmods.dev/instructions/aisa-group/posttrainbench/agents-md)

Your own site

<a href="https://agentmods.dev/instructions/aisa-group/posttrainbench/agents-md"><img src="https://agentmods.dev/badge/instructions/aisa-group/posttrainbench/agents-md/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/instructions/aisa-group/posttrainbench/agents-md"><img src="https://agentmods.dev/badge/instructions/aisa-group/posttrainbench/agents-md.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 8,543 This file is loaded in full into every session.

When invoked 8,543 The same file — it is already loaded in full.

Security scan D 3 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.08543	$0.08543
Opus 5.5	$0.03417	$0.03417
Sonnet 5.5	$0.01709	$0.01709
Haiku 4.5	$0.00854	$0.00854

Measured 26d ago against content hash cf756b6960f0, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade D, and why

PostTrainBench AGENTS.md scanned grade D with 3 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 26d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Asks for rootmediumPrivilege escalation

A mod that escalates privileges can change anything on the machine, not only the project.

chmod 600 agents/cursor_cli/cursor_auth.json

Downloads and executes remote codehighSupply chain

curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.

curl -fsS https://cursor.com/install | bash # installs to ~/.local/bin/{agent,cursor-agent}

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

via the official curl installer (`cursor.com/install`, drops `agent` + `cursor-agent` symlinks in

AGENTS.md · 480 lines

How it starts

The opening of the file, as written. The whole thing — 480 lines — stays where its author put it; the contents beside it link to each section on GitHub.

AGENTS.md

Guidelines for AI agents working on the PostTrainBench codebase.

Project Overview

PostTrainBench is a benchmark framework that measures AI agents' ability to perform LLM post-training - improving pre-trained small language models on specific benchmarks through automated research. Agents are typically given 10 hours on an H100 GPU to improve base models, with performance measured by benchmark scores.

Directory Structure

PostTrainBench/
├── agents/              # Agent implementations (claude, codex, gemini, opencode, qwen3max, glm5, ...)
├── cluster/             # Cluster-specific helper artifacts
├── containers/          # Apptainer/Singularity container definitions and helpers
├── dev_utils/           # Development/debugging utilities (run failure triage, trace extraction, ...)
├── scripts/             # Result aggregation and analysis
├── src/
│   ├── baselines/             # Baseline score computation
│   ├── commit_utils/          # HTCondor job submission utilities (incl. set_env_vars.sh)
│   ├── harbor_adapter/        # Runs the benchmark on Modal via Harbor, for external users (see "Harbor Adapter")
│   ├── judges/                # Reward-hacking judges (one folder per judge; see Safety)
│   ├── eval/
│   │   ├── general/           # Prompt generation (get_prompt.py, prompt.txt)
│   │   ├── tasks/             # Evaluation benchmarks (aime2025, gsm8k, ...)
│   │   ├── templates/         # Chat templates (Jinja2)
│   │   └── run_final_eval.sh  # The final evaluation of a trained model
│   ├── trace_parsing/         # Per-agent trace parsers (claude/codex/gemini/opencode)
│   ├── utils/                 # Utility scripts (check_cuda, system_monitor, timestamp_lines, ...)
│   └── run_task.sh            # Main task execution orchestrator
└── results/             # Evaluation results storage (path controlled by POST_TRAIN_BENCH_RESULTS_DIR)

Key Files

File	Purpose
`src/run_task.sh`	Main task execution orchestrator (runs agent, then 4 judges, then evaluation)
`src/eval/run_final_eval.sh`	The final evaluation (seeds, max-tokens retry cascade, mean into `metrics.json`); used by `run_task.sh`, `dev_utils/test_evaluation/run_only_evaluation.sh` and `scripts/rerun_eval_n_times.sh`; `--single-seed` runs one seed (`scripts/rerun_final_eval_parallel.py`)
`scripts/rerun_final_eval_parallel.py`	Reruns the final evaluation of many result dirs, one GPU job per (run, seed); submit/status/retry
`src/commit_utils/commit.sh`	Batch job submission across agents × benchmarks × models
`src/commit_utils/set_env_vars.sh`	Sources `.env` and exports `POST_TRAIN_BENCH_*` env vars
`src/commit_utils/single_task.sub`	HTCondor submission template
`src/commit_utils/single_task_gemini.sub`	Gemini-specific HTCondor submission template
`src/eval/general/get_prompt.py`	Generates agent prompts
`src/eval/general/prompt.txt`	Agent prompt template
`src/trace_parsing/parse_trace.py`	Dispatches to per-agent parser to produce human-readable trace
`src/utils/update_agent_cli.sh`	Auto-updates an agent's CLI harness to latest and records its version
`src/judges/run_judges.sh`	Runs judges on an existing result dir (`--judges` to select a subset); each writes its own per-judge JSON
`src/eval/per_sample_seed.py`	Per-sample generation seeds for every `evaluate_final_eval.py` (never one seed for all requests)
`src/utils/grant_access.py`	Gives the users in `POST_TRAIN_BENCH_USERS_ACCESS` ACL access to result dirs (run on exit by `run_task.sh`)
`src/judges/get_judge_prompt.py`	Generates judge prompts (`--judge <judge_name>`)
`containers/standard.def`	Main container definition (other `.def` files exist per-agent)
`scripts/constants.py`	Agent/benchmark mappings
`example.env`	Template for the `.env` file (API keys + `POST_TRAIN_BENCH_*` paths)

Read the full file on GitHub · 480 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

26d ago Changed · +124 lines · +2,844 tokens per session cf756b6960f0
28d ago Changed · +14 lines · +396 tokens per session 916896638d41
1mo ago First seen · 342 lines · 5,303 tokens per session scan D e2e3c8e5fc32

Subscribe to this mod's changes

PostTrainBench AGENTS.md is an instructions file published in the GitHub repository aisa-group/PostTrainBench (609 stars, last pushed 25d ago), licensed MIT. It adds 8,543 tokens to every session, about $0.0342 per session on Opus 5.5. A static security scan graded it D with 3 findings (asks for root, downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

AI Safety Claude Code Post-Training ai-research-automation codex-cli gemini-cli

vscode buildNext.instructions.md

Working notes and architecture documentation for the new esbuild-based build system in build/next. Use when making changes to the new build pipeline (transpile/bundle commands, NLS plugin, source-map handling, resource copying, or self-hosting watch tasks).

microsoft/vscode · 9,715 tokens

PostTrainBench: Instructions file for Codex