Borrowing it
Nothing to install: this file belongs to qaml-ai/camelAI. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/qaml-ai/camelAI/main/.agents/skills/running-agent-evals/SKILL.mdgit clone --depth 1 https://github.com/qaml-ai/camelAIWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/qaml-ai/camelai/running-agent-evals)<a href="https://agentmods.dev/skills/qaml-ai/camelai/running-agent-evals"><img src="https://agentmods.dev/badge/skills/qaml-ai/camelai/running-agent-evals/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/qaml-ai/camelai/running-agent-evals"><img src="https://agentmods.dev/badge/skills/qaml-ai/camelai/running-agent-evals.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 1 finding, up to high
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- high Privilege Escalation · line 122 Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00072 | $0.01885 |
| Opus 5 | $0.00036 | $0.00942 |
| Sonnet 5 | $0.00014 | $0.00377 |
| Haiku 4.5 | $0.00007 | $0.00188 |
Grade A, and why
running-agent-evals scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
curl -s https://evals.camelai.dev/api/runs?limit=20 \ How it starts
The opening of the file, as written. The whole thing — 133 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Running agent evals
Evals run locally in this repo (they need Docker + a .dev.vars); the shared history lives at
https://evals.camelai.dev (read-only viewer + upload API, behind Cloudflare Access). The full,
always-current reference is served by the viewer itself at GET /skill.
Run an eval
bun run test:eval <eval-id> # ids: workers/main/tests/evals/manifest.json
bun run test:eval:dashboard # or :deploy / :sandbox shortcuts
# custom prompt (the generic harness):
CUSTOM_EVAL_PROMPT="Build a dashboard from fake data." bun scripts/run-agent-eval.mjs custom-prompt-live
Knobs: --model <id>, --timeout-ms <ms>, EVAL_REAL_DEPLOY=0/1, CUSTOM_EVAL_* (see
bun scripts/run-agent-eval.mjs --help).
CHIRIDION_DEV_VARS_PATH / .dev.vars are read only for eval-relevant Cloudflare Access,
Cloudflare API, and judge gateway credentials/settings; ordinary eval knobs such as EVAL_MODEL
and EVAL_REPORT should be passed explicitly in the shell or CLI.
When scripts/run-eval-suite.sh runs a list or all, it automatically mints one
EVAL_BATCH_ID and default EVAL_BATCH_LABEL for the whole invocation. Pre-set those env vars to
join a run into an existing dashboard batch.
Captured artifacts use openai/gpt-5.6-luna on the compat route as the primary rollout judge
when Cloudflare AI Gateway credentials are available. The judge is blind to machine verdicts and
target-model identity, grades the task rubric from rollout/final-state evidence, and writes the
authoritative result to grading. Machine criteria remain diagnostic evidence; true harness and
artifact-contract failures still fail the run. Set EVAL_LLM_JUDGE=0 to use machine-check fallback,
or override with EVAL_JUDGE_MODEL, EVAL_JUDGE_GATEWAY_PROVIDER, or
EVAL_JUDGE_REASONING_EFFORT.
Add or update an eval
Committed evals are listed in workers/main/tests/evals/manifest.json. Each entry requires
kind: use unit for a one-mechanism check and skill for end-to-end agent ability. Keep
scorecard budgets aligned with the dashboard weighting convention: unit evals 1-5 pts, skill evals
6-20 pts scaled to task complexity. New evals should emit a task-specific rubric with 3-8
criteria whose positive weights total 100, a pass threshold (normally 75), and critical criteria
where applicable. Machine pass/fail and scorecard checks are evidence, not the primary grade. Use
optional tier: "hard" for
high-difficulty evals and optional realDeploy: true when the eval requires the testing-grounds
deploy path.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 133 lines · 72 tokens per session scan A ddea1811db2c
running-agent-evals is a skill published in the GitHub repository qaml-ai/camelAI (367 stars, last pushed 5d ago), licensed MIT. It adds 72 tokens to every session and 1,885 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
research-engineer
An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.
tika-eval-compare
Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".
neuron-evaluation-engineer
Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…
jetson-validate-image
Use after jetson-flash-image to run static BSP checks, on-target smoke/regression tests on a flashed DUT, or both. Not for build or flash steps. Triggers: validate bsp, on-target validation.
atmos-validation
Validate Atmos projects, components, arbitrary JSON Schema inputs, EditorConfig, and GitHub Actions; use affected-file selection and native CI annotations.
skill-benchmark
Benchmark AI skill effectiveness by measuring implementation quality against legacy constraints.