Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/jxoesneon/ciel/agent-evalnpx skills add jxoesneon/Ciel --skill agent-evalgit clone --depth 1 https://github.com/jxoesneon/CielWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/jxoesneon/ciel/agent-eval)<a href="https://agentmods.dev/skills/jxoesneon/ciel/agent-eval"><img src="https://agentmods.dev/badge/skills/jxoesneon/ciel/agent-eval.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00024 | $0.00835 |
| Opus 5 | $0.00012 | $0.00417 |
| Sonnet 5 | $0.00005 | $0.00167 |
| Haiku 4.5 | $0.00002 | $0.00084 |
Grade A, and why
agent-eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 5d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 77 lines — stays where its author put it; the contents beside it link to each section on GitHub.
CIEL ADAPTATION: Agent-Eval (Benchmarking Suite)
This skill formalizes the process for objectively benchmarking different coding agents or underlying models against CIEL's codebase. It replaces subjective "vibes-based" comparisons with reproducible, data-driven task evaluations.
Integration Context
Adapted from ~/.agents/skills/agent-eval/. While eval-harness is used to evaluate the code output of a specific task during active development (EDD), agent-eval is a meta-tool used by the Orchestrator to select or upgrade the agents themselves.
Core Concepts
Task Definitions
Evaluations must be defined declaratively (e.g., in YAML) specifying:
- The base repository and a pinned
commitSHA (for strict reproducibility). - The target files and the specific prompt to feed the agent.
- Deterministic judge commands (e.g.,
npm test,pytest).
Worktree Isolation
To prevent test runs from corrupting the base repository or interfering with each other, every agent evaluation run MUST execute inside an isolated Git Worktree.
🛡️ CRITICAL SAFETY MANDATE: Sandboxing & Isolation
Logical isolation (Git Worktrees) is NOT security isolation.
Evaluating autonomous agents involves executing arbitrary code generated by those agents. Running benchmarks directly on the host OS is strictly prohibited if the agents under evaluation have run_shell_command or similar capabilities.
Mandatory Sandbox Requirements
- Containerization: All evaluation trials MUST be executed within a hardened container (e.g., Docker, Podman) or a disposable Virtual Machine.
- Network Restrictions: Containers should have restricted or zero egress to prevent data exfiltration during benchmarks.
- Filesystem Mounts: Use read-only mounts for base code and ephemeral volumes for agent writes.
Evaluation Metrics
When benchmarking agents, the Orchestrator tracks the following matrix:
- Pass Rate: Did the agent produce code that passes the deterministic judge?
- Cost: API spend per task execution.
- Time: Wall-clock seconds to completion.
- Consistency: Pass rate across repeated runs (e.g., executing the same task 3 times to measure non-determinism).
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 5d ago First seen · 77 lines · 24 tokens per session scan A 34cbcb7eb409
agent-eval is a skill published in the GitHub repository jxoesneon/Ciel (1 stars, last pushed 5d ago), licensed Apache-2.0. It adds 24 tokens to every session and 835 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
redpolicy-guide
Ред-политика для гайдов по вебинарам. Превращаем транскрибации вебинаров в структурированные самостоятельные тексты. Две части: быстрый старт + подробный разбор.
Code Quality Review
Conduct a comprehensive code quality review focused on clean code, architecture, and project standards.
write-overview
Полный цикл создания обзорной статьи-эссе по теме проекта.
redpolicy-article
Ред-политика для статей-подборок сервисов. Пишем обзоры сервисов, инструментов и продуктов для аудитории проекта. Каждый сервис описывается по шаблону: название, ссылка, стоимость, описание.
surge
Use when a user provides a PRD, spec, or detailed requirements document and needs a full project delivered through iterative expert orchestration — multi-round analyze/research/design/implement/QA cycles with convergence detection. NOT for: single-file edits, quick prototypes, simple Q&A, or tasks without a written…
boil-ocean
Use when the user hands over a big multi-sprint task to finish end-to-end without supervision — "build the whole spec", "implement it all", "do all the sprints", "boil the ocean", "take the decisions yourself", "don't stop until it's done" — or when resuming an existing run (.ocean/state.json exists in the repo, or…