What is eval-reports?

Instructions for running and reporting agent evaluations in a local checkout of camelAI. An agent evaluation is a repeatable test of how an AI agent performs a task.

Use it to run a named evaluation, test a custom prompt, optionally deploy a test app, and report the run’s results.

It explains how to run the tests locally, handle required environment settings, and share results through the project’s read-only dashboard and API. It also supports custom evaluation prompts.

eval-reports — Skill for Claude Code and Codex by qaml-ai/camelAI

Install

Getting it into your agent

It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.

Clone the repo

git clone --depth 1 https://github.com/qaml-ai/camelAI

agentmods

npx agentmods add skills/qaml-ai/camelai/eval-reports

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/qaml-ai/camelai/eval-reports/github.svg)](https://agentmods.dev/skills/qaml-ai/camelai/eval-reports)

Your own site

<a href="https://agentmods.dev/skills/qaml-ai/camelai/eval-reports"><img src="https://agentmods.dev/badge/skills/qaml-ai/camelai/eval-reports/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/qaml-ai/camelai/eval-reports"><img src="https://agentmods.dev/badge/skills/qaml-ai/camelai/eval-reports.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 0 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,616 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 1 finding. A grade says what 26 rules found in the file — not that it is safe. Third-party audits

NVIDIA SkillSpector warn 7 Sept 2026

SkillSpector: 2 findings, up to high

These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →

high Privilege Escalation · line 56
Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.
high Privilege Escalation · line 63
Code accesses credential files (SSH keys, AWS credentials, etc.). This could indicate credential theft attempts.

Fix: Remove references to credential paths. Use environment variables or secrets managers. For docs, use placeholder paths (e.g., /path/to/config). Never load .env or token files in production code paths.

How audits are shown

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00000	$0.01616
Opus 5.5	$0.00000	$0.00646
Sonnet 5.5	$0.00000	$0.00323
Haiku 4.5	$0.00000	$0.00162

Measured 1mo ago against content hash 48bcce7852c4, method: parsed. Prices are Anthropic first-party input rates as of 2026-10-07, from the pricing page.

Security

Grade A, and why

eval-reports scanned grade A with 1 finding against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.

The scan reads SKILL.md. This mod also ships 11 executable files (app/lib/api.ts, app/lib/batches.ts, app/lib/format.ts, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Makes network callslowCapability

Not a fault in itself. Listed so you know the mod talks to something, and to what.

`curl -H "CF-Access-Client-Id: ..." -H "CF-Access-Client-Secret: ..."` both use these.

workers/eval-reports/SKILL.md · 114 lines

How it starts

The opening of the file, as written. The whole thing — 114 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Running agent evals

camelAI agent evals run locally in a qaml-ai/camelAI checkout — there is no remote runner. This service (https://evals.camelai.dev, workers/eval-reports) is only the shared results history: a read-only dashboard + JSON API plus an upload endpoint the local reporter uses. Everything is behind Cloudflare Access.

This document is served by the service itself at GET /skill, so it always matches the running API. (You're reading it because you fetched that endpoint.)

Run an eval (locally, in chiridion-app)

Requirements: Docker, Bun, and a .dev.vars in the repo root (the eval secret bundle).

bun run test:eval <eval-id>          # ids from workers/main/tests/evals/manifest.json
bun run test:eval:dashboard          # or :deploy / :sandbox shortcuts

# Custom prompt (not committed to the tree) via the generic harness:
CUSTOM_EVAL_PROMPT="Build a dashboard from fake data." \
  bun scripts/run-agent-eval.mjs custom-prompt-live

Common knobs: --model <id>, --timeout-ms <ms>, EVAL_REAL_DEPLOY=0/1 (publish apps to the testing-grounds namespace for real), CUSTOM_EVAL_PROJECT, CUSTOM_EVAL_REQUIRED_TRANSCRIPT_SUBSTRINGS. See bun scripts/run-agent-eval.mjs --help.

Suite and matrix invocations automatically mint a shared EVAL_BATCH_ID plus a default EVAL_BATCH_LABEL, so their member runs group together on the dashboard. Pre-set either env var when an orchestrator needs to join runs into an existing batch. Solo bun run test:eval <id> runs stay batchless and render as singleton batches.

Report a run here

Set EVAL_REPORT=1 on the run and the reporter (scripts/report-eval-run.mjs) uploads the output log and run metadata when the eval finishes — pass or fail. It also uploads the transcript artifact, including its scorecard, when the eval emitted it:

EVAL_REPORT=1 bun run test:eval dashboard-fake-data-live

Reporting is best-effort and never fails the eval. Artifactless harness failures are still reported; ingest synthesizes an evaluation_contract failure so they remain visible. Finalization is retried up to three times with the same run id before the reporter gives up. Re-report an artifact by hand:

Read the full file on GitHub · 114 lines

Files

What ships with it

42 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

1mo ago First seen · 114 lines · 0 tokens per session scan A 48bcce7852c4

Subscribe to this mod's changes

eval-reports is a skill published in the GitHub repository qaml-ai/camelAI (390 stars, last pushed today), licensed MIT. It costs nothing until one of its globs matches a file; then it loads 1,616 tokens. A static security scan graded it A with 1 finding (makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

cua-driver

Use Cua Driver for desktop or browser tasks that are awkward or unavailable through Bash/APIs, or when the user explicitly wants GUI interaction: app testing, visual bug reproduction, form filling, calendar entry, screenshots, and demo recording. Also covers Cua setup; not OpenAI Codex Computer Use or web research.

davidondrej/skills · 68 tokens

eval-reports