autoresearch

Autoresearch Skill

When to Activate

Explicit Triggers

User says any of: "optimize", "improve this", "run experiments", "find best technique", "ratchet", "hill-climbing", "prompt optimization", "autoresearch", "experiment tracking", "measure performance", "tune hyperparameters", "iterate on", "A/B test", "champion-challenger"

Implicit Signals

• User describes a problem with a measurable outcome but no clear solution path
• User wants to improve something that has been working but could be better
• User mentions comparing options, iterating, or testing variations
• User is writing prompts, code, configs, or content and wants the best version

When NOT to Use

• One-off tasks with no repeatable evaluation ("write a greeting email")
• Problems with no measurable metric ("make this nicer")
• Time-critical fixes where experimentation delays matter ("production is down")
• User explicitly says "just pick one" or "I don't care about optimal"

Data Safety Boundary

Use synthetic, non-sensitive data only; examples and tests use synthetic data only. Never send PHI, patient identifiers, PHI-bearing prompts or model responses, clinical records, secrets, or production datasets to this MCP server, evaluators, review tooling, logs, fixtures, or CI.

Core Philosophy

Autoresearch is iterative improvement against a repeatable evaluation. The loop is:

Discover → Suggest → Scaffold → Baseline → Run → Evaluate → Log → Ratchet

Ratchet principle: Only keep improvements. The best-so-far (champion) is replaced only by something measurably better.

Decision Tree: Which Technique?

Start here. Answer these questions in order.

Q1: Do you have a scalar metric?

A single number that defines success (accuracy, latency, score, cost, conversion rate).

YES → Use a ratchet pattern (single-ratchet, champion-challenger, two-loop) NO → Use an evaluator-first approach (llm-as-judge, rubric-scorer, human-approval-gate)

Q2: How long can one experiment take?