Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/richfrem/agent-plugins-skills/os-eval-runnernpx skills add richfrem/agent-plugins-skills --skill os-eval-runnergit clone --depth 1 https://github.com/richfrem/agent-plugins-skillsWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/richfrem/agent-plugins-skills/os-eval-runner)<a href="https://agentmods.dev/skills/richfrem/agent-plugins-skills/os-eval-runner"><img src="https://agentmods.dev/badge/skills/richfrem/agent-plugins-skills/os-eval-runner.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00064 | $0.00942 |
| Opus 5 | $0.00032 | $0.00471 |
| Sonnet 5 | $0.00013 | $0.00188 |
| Haiku 4.5 | $0.00006 | $0.00094 |
Grade A, and why
os-eval-runner scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 73 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Skill Improvement Evaluator
Stateless evaluation engine that scores and gates skill improvement iterations using headless Python evaluation scripts.
Ownership Boundary (Critical)
What os-eval-runner owns (permanent, version-controlled with this skill)
- Scoring scripts:
./scripts/evaluate.py,./scripts/eval_runner.py - Scaffold script:
./scripts/init_autoresearch.py - Templates:
./assets/templates/autoresearch/(program, evals, results, proposer prompt)
What lives with the target (deployed per experiment)
All experiment state deploys alongside the target (e.g. <experiment-dir>/references/program.md, <experiment-dir>/evals/evals.json, <experiment-dir>/evals/results.tsv). You MUST read the spec from <experiment-dir>/references/program.md and NOT fall back to engine-local config templates.
Phase 0: Intake Interview
Run this interview before starting any loop or evaluation. If enough information is provided in the initial prompt, skip the redundant questions.
- Q1 — What target skill are you evaluating? (Provide path to skill folder)
- Q2 — Where should the experiment files live? (Defaults to target skill directory)
- Q2b — What metric are you optimizing? (quality_score, f1, precision, recall, or heuristic)
- Q3 — What mode? (Loop mode for autonomous improvement vs QA mode for single diff validation)
- Q4 — (Loop mode) How many iterations? (Default: NEVER STOP)
- Q5 — Does evals.json exist? (If missing, scaffold from template)
- Q6 — Does program.md exist? (If missing, scaffold from template)
- Q7 — Does a baseline score exist? (If missing, run evaluate.py with
--baseline)
Two Modes: Summarized
- Mode 1: Autoresearch Loop: Autonomous iterative improvement. The agent identifies failure types, requests mutations via external proposer CLI (Copilot/Gemini), and runs the eval gate iteratively until the budget or target score is met.
- Mode 2: Single-shot QA: Simple gate validation. Evaluates one specific proposed diff against the baseline and decides KEEP (exit 0) or DISCARD (revert, exit 1).
What ships with it
37 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- assets/templates/autoresearch/copilot_proposer_prompt.md.template 80 B
- assets/templates/autoresearch/evals.json.template 64 B
- assets/templates/autoresearch/program.md.template 64 B
- assets/templates/autoresearch/results.tsv.template 65 B
- assets/templates/eval-instructions.template.md 58 B
- evals.json 759 B
- evals/evals.json 1.3 KB
- evals/results.tsv 1.7 KB
- QUICKSTART.md 4.2 KB
- README.md 3.0 KB
- references/acceptance-criteria.md 42 B
- references/autoresearch-architecture.md 48 B
- references/autoresearch-overview.md 44 B
- references/autoresearch-program-md-overview.md 55 B
- references/cheapest_models.json 40 B
- references/cheapest_models.md 38 B
- references/diagrams/autoresearch-loop.mmd 53 B
- references/diagrams/mapping-karpathy-to-skill-improvement-eval.mmd 78 B
- references/lab-space-protocol.md 41 B
- references/memory/post_run_survey.md 48 B
- references/mode-1-loop-protocol.md 43 B
- references/mode-2-qa-protocol.md 41 B
- references/overfitting-gate.md 39 B
- references/program.md 30 B
- references/quickstart-setup.md 39 B
- references/research/karpathy-autoresearch-3-file-eval.md 73 B
- references/research/meta-harness-lee-2026.md 56 B
- references/research/optimizer-engine-patterns.md 60 B
- references/survey-protocol.md 38 B
- references/troubleshooting.md 38 B
- requirements.txt 188 B
- scripts/eval_runner.py 31 B runs code
- scripts/evaluate.py 28 B runs code
- scripts/generate_milestone.py 38 B runs code
- scripts/init_autoresearch.py 37 B runs code
- scripts/plot_eval_progress.py 38 B runs code
- v2-formula-section.md 2.4 KB
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 73 lines · 64 tokens per session scan A be78ba4c14e0
os-eval-runner is a skill published in the GitHub repository richfrem/agent-plugins-skills (6 stars, last pushed yesterday), licensed MIT. It adds 64 tokens to every session and 942 once invoked, about $0.0003 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
composio
Use Composio from Agent Swarm through the agent-swarm x composio CLI route, the swarmx MCP tool, or a registered ctx.api.composio script connection. Trigger when a task needs connected third-party app tools such as Gmail, Google Calendar, Google Docs, Google Drive, GitHub, Slack, Notion, or HubSpot through Tool Router…
soul
Embody this digital identity. Read SOUL.md first, then STYLE.md, then examples/. Become the person—opinions, voice, worldview.
close-issue
Close a GitHub or GitLab issue with a summary comment.
implement-issue
Implement a GitHub issue or GitLab issue and create a PR/MR.
bbc-skill
Fetch Bilibili (哔哩哔哩) video comments for UP主 self-analysis. Use when the user asks to collect, download, export, or analyze comments on a Bilibili video (BV号 / URL / UID). Produces JSONL + summary.json suitable for further Claude Code analysis (sentiment, keywords, audience trends). Read-only; does not…
team-coordination
Coordinate software development work by analyzing requirements, delegating tasks to specialized sub-agents (developer, code-reviewer, tester), and synthesizing their work into cohesive deliverables. Use this for complex projects that require multiple specialized perspectives.