recursive-improve

recursive-improve: Agent Improvement Pipeline

End-to-end pipeline: trace analysis → skill extraction → domain context → metrics → rubric → action plan → review → fixes.

Prerequisites

Traces must exist in eval/traces/. If they don't:

• Ask the user for their traces directory
• Copy .json, .md, and .toon files into eval/traces/

Skip condition: If eval/stage1_insights_summary.md already exists (from a prior run or from recursive-improve analyze), skip Stages 0 and 1 — go directly to Stage 2.

---

Stage 0: Trace Analysis

Analyze raw execution traces to extract learnings. This stage adapts ACE's recursive reflector methodology — a structured 6-phase strategy that moves from data discovery through verified deep-dives to synthesized, evidence-backed insights.

Inputs

• eval/traces/ — raw trace files (.json, .md, .toon)

Phase 1: Discover

Map the data shape and inventory. Do NOT judge outcomes yet — just catalog what you have.

1. Read 2-3 trace files. Identify:

   • Top-level keys and message schema (3 levels deep)
   • Message format: role, content, tool_calls, turn_idx, etc.
   • Total trace count and per-trace message counts

2. Search for agent operating rules, policy, or instructions embedded in the traces — these are often in large strings (>500 chars). Check:

   • role: "system" messages
   • info.environment_info.policy or similar fields
   • Large embedded strings in any field

3. Build an inventory table:

   File                  Messages  Has system prompt?  Has tool calls?
   trace_001.json        42        yes                 yes
   trace_002.json        18        yes                 no
   ...
   

4. Record discovered rules/policy verbatim — understanding what the agent was supposed to do is essential for evaluating what it actually did.

Phase 2: Derive Evaluation Criteria

Based on your discovery (schema, rules, patterns), define specific evaluation criteria to apply to every trace during the survey phase.