auto-research

Auto-Research: Experiment-Driven Improvement

After a feature passes its spec requirements, use Andrej Karpathy's auto-research methodology to autonomously discover improvements. This is not unstructured brainstorming — it is a disciplined experiment loop that produces measured, defensible results.

For a CAC mission tool, this matters for two reasons:

1. Performance — investigators need answers fast. A triage pass that takes 8 hours instead of 2 means children wait longer for help.
2. Defensibility — every claimed improvement is backed by a logged, reproducible experiment. When a defense expert asks "how do you know your hash matcher is faster than the previous version?", the answer is a numbered experiment record with before / after measurements and the SHA of the branch that produced them.

The Loop

while improvements_possible:
    1. READ        — current spec thresholds + benchmark baselines
    2. HYPOTHESIZE — one concrete, testable change
    3. IMPLEMENT   — in an isolated branch or worktree
    4. MEASURE     — run benchmarks, compare to baseline
    5. EVALUATE    — keep if better, discard if not
    6. RATCHET     — update spec floor to new baseline if merged

The agent runs this loop. The investigator reviews each ratchet decision before it is merged.

Rules

1. Always Measure, Never Guess

• Every hypothesis MUST have a measurable prediction ("this change should improve throughput by ~15%" or "this should reduce false positives below 0.5%").
• Run the actual benchmarks. Do not eyeball code and declare it faster.
• Record before / after numbers with the same workload, same hardware profile, and same measurement methodology. A faster benchmark on a different machine proves nothing.

2. One Variable at a Time

• Each experiment changes exactly one thing: an algorithm, a buffer size, a parallelism strategy, a data structure, a model checkpoint, a confidence threshold.
• Do not bundle multiple changes into one experiment. If you change two things and the metric improves, you cannot attribute the improvement to either change.