Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/agno-agi/agentos-render/create-evalsnpx skills add agno-agi/agentos-render --skill create-evalsgit clone --depth 1 https://github.com/agno-agi/agentos-renderWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/agno-agi/agentos-render/create-evals)<a href="https://agentmods.dev/skills/agno-agi/agentos-render/create-evals"><img src="https://agentmods.dev/badge/skills/agno-agi/agentos-render/create-evals.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00089 | $0.01300 |
| Opus 5 | $0.00044 | $0.00650 |
| Sonnet 5 | $0.00018 | $0.00260 |
| Haiku 4.5 | $0.00009 | $0.00130 |
Grade A, and why
create-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
This is a copy
100% identical to create-evals — 0 lines differ, which has more behind it and is treated as the original. This page carries a canonical link to it rather than competing with it.
How it starts
The opening of the file, as written. The whole thing — 79 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Create Evals
Coding-agent workflow: run as
/create-evalsor by describing the task.
Turn what an agent promises into Case entries in evals/cases.py. The template's cases cover the reference components only; a user-built agent is invisible to the suite until this skill writes one. Failing suite → eval-and-improve; agent needs hardening → improve-agent.
Preconditions: Postgres on 5432, venv active (source .venv/bin/activate; ./scripts/venv_setup.sh if missing), .env populated.
Be self-driving: the repo and the database answer most questions. Ask the user only what they alone know — which jobs matter and which failures would hurt. One pick per exchange, recommendation first.
1. Pick the agent
The user's choice, or the least-covered component — almost always one of theirs. Source components have a file under agents/ or teams/. Studio-built components have none: Agent.load("<id>", db=eval_db, registry=registry, published_only=True) (or Team.load) rehydrates the published config at import time; it returns None for an archived or unpublished component, and a Case with neither field set raises — guard the load or the whole suite fails at import. Workflows cannot be case targets.
2. Map what it promises
Read the file (or the published config: eval_db.get_config(component_id="<id>")["config"]). Every "always", "never", "use X for Y" is a case. Note the tools for reliability assertions.
Two checks decide the hooks:
- Can the run reach the ungated create/edit/publish Studio tools?
StudioToolsdirectly or through a team member —platform-builderalways, and anagnocase one delegation from a build. Those take**BUILDER_HOOKS. - Does it carry learning stores (
learning=) or theshared_notestoolkit? Those take**LEARNING_HOOKS. Builder hooks are a strict superset; when in doubt, take them.
3. Mine the platform
from db import get_postgres_db
db = get_postgres_db()
sessions, _ = db.get_sessions(component_id="<agent-id>", limit=20, deserialize=False)
asks = [run["input"]["input_content"] for s in sessions for run in (s.get("runs") or []) if run.get("input")]
evals, _ = db.get_eval_runs(limit=20, deserialize=False)
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 3d ago First seen · 79 lines · 89 tokens per session scan A 026e02916d3e
create-evals is a skill published in the GitHub repository agno-agi/agentos-render (2 stars, last pushed 4d ago), licensed Apache-2.0. It adds 89 tokens to every session and 1,300 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. It is 100% identical to create-evals, differing in 0 lines, and is treated as a copy.
Other skills, from other repositories
improve-agent
Autonomous hardening loop for an existing agent — derive probes from the agent's INSTRUCTIONS and from its real usage recorded in the database, run them against the live container, judge responses, edit the agent file, and re-probe until it reliably does what its instructions say. No user input needed. Use to harden…
deploy-platform
Deploy this AgentOS to production with this template's deploy scripts — preflight the provider CLI and account, run the up.sh script, complete the JWT key step, verify the live platform on its public URL, then hand over the redeploy/logs/teardown instructions. Use this skill when the user asks to deploy, ship to…
eval-and-improve
Run the eval suite (python -m evals), diagnose every failure, fix what's in scope, and loop until all cases pass. Use when evals are failing — including overnight run-evals schedule failures — or when the user wants to run, diagnose, or repair the eval suite. To author new coverage, use create-evals instead.
setup-platform
Set up this AgentOS from a fresh clone — confirm Docker, configure .env, boot the containers, prove the MCP endpoint live, connect the AgentOS UI, then build the user's first agent. Use when the user asks to set up the platform, get started, or bring this repo up on a new machine.
create-evals
Author eval coverage for an agent in this AgentOS — map what the agent promises, mine real sessions and eval history from Postgres for scenarios, propose capabilities worth testing, then write, run, and audit Case entries in evals/cases.py. Use when the user wants evals created, coverage added, or an agent's behavior…
review-and-improve
Repo-wide drift sweep for public-readiness — diff docs against code, confirm every agent is registered and reachable, every env var documented, every doc path exists, and scripts behave as advertised; auto-fix mechanical drift and flag the rest. Use before a public release or after a refactor.