eval_writeup

/eval_writeup

End-to-end pipeline: parse a Prime Intellect eval run, read the environment source, and auto-generate a blog post from eval metrics. Use --parse-only to just extract structured metrics without writing a post. Distinct from /add_post (manually-written original blog post) — use /eval_writeup when starting from eval output, /add_post when writing from scratch.

Consolidates the former /parse_eval command (now /eval_writeup --parse-only).

Usage

/eval_writeup <eval-output-or-job-id> [--title "Custom Title"] [--parse-only]

Examples:

• /eval_writeup /path/to/eval_output.txt
• /eval_writeup swarm_economy_openai_gpt_4.1_mini_20260212_205322_b3c5c09f --title "GPT-4.1 Mini Plays the SWARM Economy"
• /eval_writeup /path/to/eval_output.txt --parse-only (just parse metrics, no blog post)
• /eval_writeup swarm_economy_openai_gpt_4.1_mini_20260212_205322_b3c5c09f --parse-only

Argument parsing

Parse $ARGUMENTS to extract:

• --parse-only: Only run Phase 1 (parse eval output into structured metrics). Skip environment reading, analysis, and blog post generation.
• --title "...": Custom blog post title (ignored in --parse-only mode).
• Remaining arg: eval output file path or job ID.

If given a file path, read it directly. If given a job ID, look for the most recent matching output in the tool results cache or ask the user to provide the output.

---

--parse-only mode

Parse raw prime eval run output into structured metrics tables. This is a read-only analysis — no files are created.

1) Extract header metadata

From the --- Evaluation --- block, extract:

• Environment: name
• Model: provider/model
• Provider: inference endpoint URL
• Examples: count
• Rollouts per example: count

2) Extract reward metrics

From the --- All --- / Rewards: section, parse each metric line:

metric_name: avg - X.XXX, std - X.XXX

Build a summary table:

Metric	Avg	Std	Min	Max
reward	0.830	0.092	0.690	1.028
payoff_reward	0.701	0.101	0.540	0.916
...	...	...	...	...