eval

Evaluate and improve one healthcare agent's system prompt. Run up to 5 iterations of: prepare fixed questions -> answer -> judge -> improve -> re-score -> commit if better.

Target agent: agents/$ARGUMENTS.md

---

Support Docs

Read these as operating instructions for the eval run:

• eval/rubric.md — frozen scoring metric.
• eval/role-baselines/$ARGUMENTS.md — frozen expected-capability baseline for this agent.
• eval/role-baselines/INDEX.md — confirms baseline coverage for all installable agents.
• docs/eval/exam-architect-playbook.md — scorer and question-writing guidance.
• docs/eval/model-tuning.md — model-role and manifest guidance for current SOTA models.
• eval/meta/README.md and linked meta-eval docs — optional calibration checks for release or close-call scoring.

Do not use or recreate the retired Python/DSPy harness. The active system is this command, the frozen rubric, role baselines, optional local run logs, and git.

---

Preferred Execution Mode

When the runtime supports native subagents, model routing, or specialist workers, prefer a four-role loop:

• Parent orchestrator — owns preflight checks, fixed-question persistence within an iteration, line-cap enforcement, local run logs, eval/results.tsv append, and commit/revert.
• Scorer/judge — strongest available reasoning model; read-only; generates the exam, scores answers, checks calibration risk, and returns structured critique.
• Editor — faster strong model; edits only agents/$ARGUMENTS.md using the scorer's brief.
• Adjudicator — optional different model family for close deltas, high-risk roles, suspicious scoring, or release scoring.

Do not recursively invoke a full agent CLI inside itself when native subagents are available. If no subagents are available, a single agent may run the same workflow end-to-end.

Use docs/eval/model-tuning.md for model-role guidance. Pin exact model IDs in run logs when the runtime exposes them; do not rely only on UI or marketing names.