What is PostTrainBench AGENTS.md?

Repository instructions for PostTrainBench, a framework that measures how well AI agents improve small language models on set benchmarks.

Use them when adding an agent, working on model post-training experiments, changing benchmark evaluation, or using the project’s job and analysis utilities.

They explain the project layout, important files, and conventions so an agent can make changes without overlooking the training, evaluation, or cluster tooling.

PostTrainBench AGENTS.md — Instructions file by aisa-group

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add instructions/aisa-group/posttrainbench/agents-md

Clone the repo

git clone --depth 1 https://github.com/aisa-group/PostTrainBench

Made for: Codex, OpenCode.

Per session 5,303 This file is loaded in full into every session.

When invoked 5,303 The same file — it is already loaded in full.

Security scan D 3 findings. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.05303	$0.05303
Opus 5	$0.02652	$0.02652
Sonnet 5	$0.01061	$0.01061
Haiku 4.5	$0.00530	$0.00530

Measured 2d ago against content hash e2e3c8e5fc32, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade D, and why

PostTrainBench AGENTS.md scanned grade D with 3 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Asks for rootmediumPrivilege escalation

A mod that escalates privileges can change anything on the machine, not only the project.

chmod 600 agents/cursor_cli/cursor_auth.json

Downloads and executes remote codehighSupply chain

curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.

curl -fsS https://cursor.com/install | bash # installs to ~/.local/bin/{agent,cursor-agent}

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

via the official curl installer (`cursor.com/install`, drops `agent` + `cursor-agent` symlinks in

AGENTS.md · 342 lines

How it starts

The opening of the file, as written. The whole thing — 342 lines — stays where its author put it; the contents beside it link to each section on GitHub.

AGENTS.md

Guidelines for AI agents working on the PostTrainBench codebase.

Project Overview

PostTrainBench is a benchmark framework that measures AI agents' ability to perform LLM post-training - improving pre-trained small language models on specific benchmarks through automated research. Agents are typically given 10 hours on an H100 GPU to improve base models, with performance measured by benchmark scores.

Directory Structure

PostTrainBench/
├── agents/              # Agent implementations (claude, codex, gemini, opencode, qwen3max, glm5, ...)
├── cluster/             # Cluster-specific helper artifacts
├── containers/          # Apptainer/Singularity container definitions and helpers
├── dev_utils/           # Development/debugging utilities (run failure triage, trace extraction, ...)
├── scripts/             # Result aggregation and analysis
├── src/
│   ├── baselines/             # Baseline score computation
│   ├── commit_utils/          # HTCondor job submission utilities (incl. set_env_vars.sh)
│   ├── judges/                # Reward-hacking judges (one folder per judge; see Safety)
│   ├── eval/
│   │   ├── general/           # Prompt generation (get_prompt.py, prompt.txt)
│   │   ├── tasks/             # Evaluation benchmarks (aime2025, aime2026, gsm8k, ...)
│   │   └── templates/         # Chat templates (Jinja2)
│   ├── trace_parsing/         # Per-agent trace parsers (claude/codex/gemini/opencode)
│   ├── utils/                 # Utility scripts (check_cuda, system_monitor, timestamp_lines, ...)
│   └── run_task.sh            # Main task execution orchestrator
└── results/             # Evaluation results storage (path controlled by POST_TRAIN_BENCH_RESULTS_DIR)

Key Files

File	Purpose
`src/run_task.sh`	Main task execution orchestrator (runs agent, then 4 judges, then evaluation)
`src/commit_utils/commit.sh`	Batch job submission across agents × benchmarks × models
`src/commit_utils/set_env_vars.sh`	Sources `.env` and exports `POST_TRAIN_BENCH_*` env vars
`src/commit_utils/single_task.sub`	HTCondor submission template
`src/commit_utils/single_task_gemini.sub`	Gemini-specific HTCondor submission template
`src/eval/general/get_prompt.py`	Generates agent prompts
`src/eval/general/prompt.txt`	Agent prompt template
`src/trace_parsing/parse_trace.py`	Dispatches to per-agent parser to produce human-readable trace
`src/utils/update_agent_cli.sh`	Auto-updates an agent's CLI harness to latest and records its version
`src/judges/run_judges.sh`	Runs judges on an existing result dir (`--judges` to select a subset); each writes its own per-judge JSON
`src/judges/get_judge_prompt.py`	Generates judge prompts (`--judge <judge_name>`)
`containers/standard.def`	Main container definition (other `.def` files exist per-agent)
`scripts/constants.py`	Agent/benchmark mappings
`example.env`	Template for the `.env` file (API keys + `POST_TRAIN_BENCH_*` paths)

Read the full file on GitHub · 342 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2d ago First seen · 342 lines · 5,303 tokens per session scan D e2e3c8e5fc32

Subscribe to this mod's changes

PostTrainBench AGENTS.md is an instructions file published in the GitHub repository aisa-group/PostTrainBench (538 stars, last pushed 11d ago), licensed MIT. It adds 5,303 tokens to every session, about $0.0265 per session on Opus 5. A static security scan graded it D with 3 findings (asks for root, downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

AI Safety Data and AI Post-Training Productivity Research ai-research-automation claude-code codex-cli gemini-cli

PostTrainBench AGENTS.md

Getting it into your agent

What it costs to keep this loaded

Grade D, and why

Asks for rootmediumPrivilege escalation

Downloads and executes remote codehighSupply chain

Makes network callslowCapability

How it starts

AGENTS.md

Project Overview

Directory Structure

Key Files

What this file has done since we first saw it

Other instructions, from other repositories

obsidian-mind CLAUDE.md

obsidian-mind AGENTS.md

TokenTracker AGENTS.md

parallel-code CLAUDE.md

neo4j-skills AGENTS.md

ox AGENTS.md