What is autoresearch?

An automated experiment loop for improving a system when you can measure its results. Separate agents generate changes, check them against pass/fail rules and held-out tests, and review how the search is going.

Use it to tune parameters, compare configurations, explore improvements, and continue searching when a single best result is not enough.

It reduces the need to run experiments and judge results by hand, while helping avoid overfitting to the tests used during tuning.

autoresearch — Skill for Claude Code and Codex by pbc-os

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add pbc-os/smb-starter-kit --skill autoresearch

Clone the repo

git clone --depth 1 https://github.com/pbc-os/smb-starter-kit

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/pbc-os/smb-starter-kit/autoresearch/github.svg)](https://agentmods.dev/skills/pbc-os/smb-starter-kit/autoresearch)

Your own site

<a href="https://agentmods.dev/skills/pbc-os/smb-starter-kit/autoresearch"><img src="https://agentmods.dev/badge/skills/pbc-os/smb-starter-kit/autoresearch/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/pbc-os/smb-starter-kit/autoresearch"><img src="https://agentmods.dev/badge/skills/pbc-os/smb-starter-kit/autoresearch.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 63 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 5,489 The whole file, excluding the scripts and references it only reads on demand.

Security scan B 1 finding. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00063	$0.05489
Opus 5.5	$0.00025	$0.02196
Sonnet 5.5	$0.00013	$0.01098
Haiku 4.5	$0.00006	$0.00549

Measured 2mo ago against content hash 7b84a634e731, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade B, and why

autoresearch scanned grade B with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Asks for rootmediumPrivilege escalation

A mod that escalates privileges can change anything on the machine, not only the project.

chmod 444 eval.py # or eval.sh, eval.js, etc.

skills/tier-5-automation/autoresearch/SKILL.md · 414 lines

How it starts

The opening of the file, as written. The whole thing — 414 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Autoresearch

Autonomous iterative improvement for any measurable system.

Built on Karpathy's autoresearch pattern, extended with:

Critic agent with binary pass/fail gates and holdout validation — generator/verifier separation so the agent can't grade its own homework
Coverage-driven exploration instead of pure hill-climbing — don't spend 40 experiments tuning one knob when another knob has never been touched
Metacognitive self-modification — a meta-reviewer agent that can improve the researcher's own optimization strategy between runs
Archive of stepping stones — maintain a diverse population of good configurations, not just the single best, so you can branch out when stuck in a local optimum

See the references/ folder for the academic and practitioner sources behind each of these extensions.

Triggers

"autoresearch"
"run autoresearch on..."
"iteratively improve..."
"optimize [X] autonomously"
"run experiments on..."
"self-improve [system]"
"find better parameters for..."
"tune [X]"
User wants to autonomously improve any measurable system

The Pattern

Three agents work in a pipeline, coordinated by a small set of files on disk:

┌─────────────────────────────────────────────────────────────────┐
│                                                                 │
│  ┌─────────────┐    ┌─────────────┐    ┌──────────────────┐     │
│  │  RESEARCHER │───▶│   CRITIC    │───▶│  META-REVIEWER   │     │
│  │  (proposes) │    │ (validates) │    │ (every N exps)   │     │
│  └─────────────┘    └─────────────┘    └──────────────────┘     │
│        │                   │                    │              │
│        ▼                   ▼                    ▼              │
│  1. Analyze weakness  4. Holdout gate      7. Review full log  │
│  2. Form hypothesis   5. Overfitting check 8. Update strategy  │
│  3. Change 1 param    6. Binary pass/fail  9. Suggest new      │
│     + run eval           → KEEP/DISCARD       dimensions       │
│        │                   │                    │              │
│        └───────────────────┴────────────────────┘              │
│                            │                                   │
│                    ┌───────▼────────┐                          │
│                    │ COVERAGE MAP   │                          │
│                    │ What's been    │                          │
│                    │ explored?      │                          │
│                    │ What's missing?│                          │
│                    └────────────────┘                          │
│                            │                                   │
│                    ┌───────▼────────┐                          │
│                    │    ARCHIVE     │                          │
│                    │ Top N configs  │                          │
│                    │ Branch from    │                          │
│                    │ diverse parents│                          │
│                    └────────────────┘                          │
└─────────────────────────────────────────────────────────────────┘

Read the full file on GitHub · 414 lines

Files

What ships with it

6 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

2mo ago First seen · 414 lines · 63 tokens per session scan B 7b84a634e731

Subscribe to this mod's changes

autoresearch is a skill published in the GitHub repository pbc-os/smb-starter-kit (10 stars, last pushed 5mo ago), licensed MIT. It adds 63 tokens to every session and 5,489 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it B with 1 finding (asks for root). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Analytics and metrics Research

autoresearch next to 6 skills from other repositories
Mod	Type	Stars	Last push	Licence	Scan	Tokens	Works with
autoresearchpbc-os/smb-starter-kit	Skill	10	5mo ago	MIT	B	63	Claude Code, Codex
imagegenopenai/codex	Skill	133k	yesterday	Apache-2.0	A	113	Codex
playwright-climicrosoft/playwright	Skill	99k	yesterday	Apache-2.0	A	19	Claude Code
local-ai-agentsmicrosoft/ai-agents-for-beginners	Skill	79k	4d ago	MIT	A	200	Claude Code, Codex
systematic-debuggingobra/superpowers	Skill	313k	3d ago	MIT	A	21	Claude Code
next-cache-components-adoptionvercel/next.js	Skill	143k	yesterday	MIT	A	95	Claude Code
next-partial-prefetching-adoptionvercel/next.js	Skill	143k	yesterday	MIT	A	103	Claude Code

autoresearch