cpv-evaluate-agent-variants

cpv-evaluate-agent-variants

Overview

Drives scripts/cpv_agent_eval.py, which compares the ORIGINAL agent against any subset of the three architectures:

Architecture	skills: lists	Skills execute in
ALL-IN-ONE	every skill it needs	the same agent
ONE-FOR-ALL	every skill it needs	a separate subagent per skill
PLUGIN-OMNI	the plugin's the-skills-menu + the companion	resolved at runtime from the menu

They trade in opposite directions — ALL-IN-ONE pays one large cache creation for turn-1 readiness, ONE-FOR-ALL pays more turns for a near-empty context per node, PLUGIN-OMNI pays a menu lookup to keep the prefix tiny — so which one wins is an empirical question about a SPECIFIC agent. Two tiers answer it, and they are kept strictly apart.

The honesty rule that governs every number: a static estimate must never be presented as a measured result. Every figure carries the tier that produced it — tier1-static (computed from real files, no model involved) or tier2-live (measured from a real run). A missing measurement is reported as UNKNOWN with a non-zero exit: never as 0, never simulated, never folded into a passing result.

Prerequisites

• uv on PATH.
• The ORIGINAL agent .md file (the baseline every delta is measured against).
• Any variant .md files to compare, normally produced by scripts/convert_agent.py --to all-in-one|one-for-all|plugin-omni.
• The skills the agents reference resolvable on disk — either inside the agent's own plugin (auto-resolved) or passed with --skills-root. A wrong root makes every preload unresolvable and the cost model vacuous, so a bad root is a hard error.
• For the live tier only: a task file and a directory of captured per-run timings.

Instructions

Tier 1 — static cost model (always runs, zero LLM calls)

1. Run the script with the ORIGINAL plus every variant file that exists, and select the rows with --variants (any subset of original,all-in-one,one-for-all,plugin-omni). "The original vs the two new versions" is --variants original,all-in-one,one-for-all.
2. Read the per-variant measurements, all taken over the real files:
   • cached-prefix tokens — the agent body PLUS the FULL content of every preloaded skill. A skills: entry injects that skill's whole SKILL.md into every invocation, so that content IS a prefix cost; listing a skill is never free.
   • injected tokens per invocation — the preload portion, i.e. what the skills: list adds on top of the body, and what would vanish if the same skill were reached at runtime instead.
   • tool-schema surface — the count of declared tools, or inherited when the agent declares no tools: field (never reported as a surface of 0).
   • closure size — files and bytes across every reachable skill's SKILL.md, references/** and scripts/**.
   • turn-1 readiness — whether every reachable skill is already in the prefix, or some arrive only after a Skill call.
   • projected N-turn prefix cost — turn 1 at the cache-write rate, each later turn at the cache-read rate.
3. Optional flags: --skills-root PATH (repeatable), --max-depth N, --turns N (repeatable), --json, --report PATH.