eval-and-improve

Eval and Improve

Coding-agent workflow: run as /eval-and-improve or by describing the task.

Run the suite, diagnose every failure, fix what's in scope, stop when green. Cases live in evals/cases.py (agno.eval.Case), the setup/teardown sweeps in evals/hooks.py, the entrypoint in evals/__main__.py. Each case uses an LLM judge (criteria), a tool-call assertion (expected_tool_calls), and/or a deterministic scorer.

0. Preconditions

• Postgres on 5432 (nc -z localhost 5432; else docker compose up -d agentos-db).
• Venv active (source .venv/bin/activate; ./scripts/venv_setup.sh if missing). No server needed — cases import the components directly.
• .env has OPENAI_API_KEY (and PARALLEL_API_KEY if you use one — it pins Agno's expected web tool name at import). Worktrees don't inherit .env.

1. Run

python -m evals --tag smoke            # fast template checks
python -m evals --tag release          # broader pre-release checks
python -m evals --tag live             # current web/source checks
python -m evals --name <case>          # one case
python -m evals --tag smoke --list     # what a selector picks, without running
python -m evals --json-output out.json # machine-readable (carries judge_reason)
python -m evals -v                     # stream the runs

Exit 0 = all passed. Be the only writer: teardowns sweep by snapshot diff, so a note or entity a teammate files during a case gets deleted. Coming from a scheduled failure? Find the case in eval history (db.get_eval_runs(), os.agno.com, or Platform Manager) and reproduce it with --name first; one that won't reproduce is usually environment. Stderr noise at the end of a run (Event loop is closed, httpx timeouts) is harmless.

2. Diagnose

Symptom	Likely cause	Fix
Judge: right answer, missing X	Instructions don't push for X	agents/<slug>.py — tighten the rule
Judge: fabricated	Should have said it didn't know	Add a "say so plainly" rule
Reliability: missing tool	Routing rule weak, or the case too narrow	Strengthen the rule, or broaden expected_tool_calls
Reliability: additional tool with allow_additional_tool_calls=False	Agent fanned out	Tighten instructions or allow it
Agno web tool name mismatch (parallel_search ↔ web_search)	PARALLEL_API_KEY differs between .env and shell	Sync and re-run
Flips PASS/FAIL across runs, no change	Judge variance	Re-run 2–3×; still flipping → make criteria more falsifiable
Fails in the suite, passes alone	Transient flake / 429	Re-run alone, then the suite; persistent 429s → back off
Many fail at once	Broad regression (model, MCP down, tool removed)	Find the root cause; no prompt edits
run paused awaiting user input	Hit a HITL gate (Builder archive/delete) — never graded	Keep the case input on the ungated side
cleanup: in the error	A teardown couldn't delete what the case created	Hard-delete by id (eval_db.delete_component(id, hard_delete=True), ScheduleManager(eval_db).delete(id), eval_db.delete_learning(id), notes.delete(path)); don't touch the agent or case
refusing to sweep …	A guard tripped: rows predate the snapshot, or more rows than a case plausibly creates	Don't loosen the guard; inspect, delete the case's rows by hand, re-run