Borrowing it
Nothing to install: this file belongs to agno-agi/agentos-modal. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/agno-agi/agentos-modal/main/.agents/skills/eval-and-improve/SKILL.mdgit clone --depth 1 https://github.com/agno-agi/agentos-modalWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/agno-agi/agentos-modal/eval-and-improve)<a href="https://agentmods.dev/skills/agno-agi/agentos-modal/eval-and-improve"><img src="https://agentmods.dev/badge/skills/agno-agi/agentos-modal/eval-and-improve/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/agno-agi/agentos-modal/eval-and-improve"><img src="https://agentmods.dev/badge/skills/agno-agi/agentos-modal/eval-and-improve.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00074 | $0.01491 |
| Opus 5 | $0.00037 | $0.00745 |
| Sonnet 5 | $0.00015 | $0.00298 |
| Haiku 4.5 | $0.00007 | $0.00149 |
Grade A, and why
eval-and-improve scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
100% identical to eval-and-improve — 0 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 79 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Eval and Improve
Coding-agent workflow: run as
/eval-and-improveor by describing the task.
Run the suite, diagnose every failure, fix what's in scope, stop when green. Cases live in evals/cases.py (agno.eval.Case), the setup/teardown sweeps in evals/hooks.py, the entrypoint in evals/__main__.py. Each case uses an LLM judge (criteria), a tool-call assertion (expected_tool_calls), and/or a deterministic scorer.
0. Preconditions
- Postgres on 5432 (
nc -z localhost 5432; elsedocker compose up -d agentos-db). - Venv active (
source .venv/bin/activate;./scripts/venv_setup.shif missing). No server needed — cases import the components directly. .envhasOPENAI_API_KEY(andPARALLEL_API_KEYif you use one — it pins Agno's expected web tool name at import). Worktrees don't inherit.env.
1. Run
python -m evals --tag smoke # fast template checks
python -m evals --tag release # broader pre-release checks
python -m evals --tag live # current web/source checks
python -m evals --name <case> # one case
python -m evals --tag smoke --list # what a selector picks, without running
python -m evals --json-output out.json # machine-readable (carries judge_reason)
python -m evals -v # stream the runs
Exit 0 = all passed. Be the only writer: teardowns sweep by snapshot diff, so a note or entity a teammate files during a case gets deleted. Coming from a scheduled failure? Find the case in eval history (db.get_eval_runs(), os.agno.com, or Platform Manager) and reproduce it with --name first; one that won't reproduce is usually environment. Stderr noise at the end of a run (Event loop is closed, httpx timeouts) is harmless.
2. Diagnose
| Symptom | Likely cause | Fix |
|---|---|---|
| Judge: right answer, missing X | Instructions don't push for X | agents/<slug>.py — tighten the rule |
| Judge: fabricated | Should have said it didn't know | Add a "say so plainly" rule |
| Reliability: missing tool | Routing rule weak, or the case too narrow | Strengthen the rule, or broaden expected_tool_calls |
Reliability: additional tool with allow_additional_tool_calls=False |
Agent fanned out | Tighten instructions or allow it |
Agno web tool name mismatch (parallel_search ↔ web_search) |
PARALLEL_API_KEY differs between .env and shell |
Sync and re-run |
| Flips PASS/FAIL across runs, no change | Judge variance | Re-run 2–3×; still flipping → make criteria more falsifiable |
| Fails in the suite, passes alone | Transient flake / 429 | Re-run alone, then the suite; persistent 429s → back off |
| Many fail at once | Broad regression (model, MCP down, tool removed) | Find the root cause; no prompt edits |
run paused awaiting user input |
Hit a HITL gate (Builder archive/delete) — never graded | Keep the case input on the ungated side |
cleanup: in the error |
A teardown couldn't delete what the case created | Hard-delete by id (eval_db.delete_component(id, hard_delete=True), ScheduleManager(eval_db).delete(id), eval_db.delete_learning(id), notes.delete(path)); don't touch the agent or case |
refusing to sweep … |
A guard tripped: rows predate the snapshot, or more rows than a case plausibly creates | Don't loosen the guard; inspect, delete the case's rows by hand, re-run |
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 79 lines · 74 tokens per session scan A 43455a1b1ede
eval-and-improve is a skill published in the GitHub repository agno-agi/agentos-modal (2 stars, last pushed 9d ago), licensed Apache-2.0. It adds 74 tokens to every session and 1,491 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. It is 100% identical to eval-and-improve, differing in 0 lines, and is treated as a copy.
Other skills, from other repositories
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
tika-eval-compare
Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".
jetson-validate-image
Use after jetson-flash-image to run static BSP checks, on-target smoke/regression tests on a flashed DUT, or both. Not for build or flash steps. Triggers: validate bsp, on-target validation.
atmos-validation
Validate Atmos projects, components, arbitrary JSON Schema inputs, EditorConfig, and GitHub Actions; use affected-file selection and native CI annotations.
skill-benchmark
Benchmark AI skill effectiveness by measuring implementation quality against legacy constraints.
restore-internals-seams-in-finally-blocks-after-each-test
When delegating a task affected by this skill, include.