Borrowing it
Nothing to install: this file belongs to OctopusGarage/alcove. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/OctopusGarage/alcove/main/.claude/commands/eval-ai.mdgit clone --depth 1 https://github.com/OctopusGarage/alcoveWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/commands/octopusgarage/alcove/eval-ai)<a href="https://agentmods.dev/commands/octopusgarage/alcove/eval-ai"><img src="https://agentmods.dev/badge/commands/octopusgarage/alcove/eval-ai/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/commands/octopusgarage/alcove/eval-ai"><img src="https://agentmods.dev/badge/commands/octopusgarage/alcove/eval-ai.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00018 | $0.00344 |
| Opus 5 | $0.00009 | $0.00172 |
| Sonnet 5 | $0.00004 | $0.00069 |
| Haiku 4.5 | $0.00002 | $0.00034 |
Grade A, and why
eval-ai scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 10d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
What it actually says
Run Alcove's AI quality eval from the repository root.
This eval is separate from deterministic smoke. It reruns smoke suites, builds an AI review packet, then asks an AI reviewer to judge usefulness, intent fit, module consistency, and agent-facing quality.
Command:
repo="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
cd "$repo"
scripts/eval-ai.sh
Focused eval can refresh only selected suites:
ALCOVE_AI_EVAL_SUITES=isolated,mcp_matrix ALCOVE_AI_EVAL_PROVIDER=none ALCOVE_AI_EVAL_RUN_CHECK=0 scripts/eval-ai.sh
ALCOVE_AI_EVAL_SUITES=isolated,mcp_matrix ALCOVE_AI_EVAL_SKIP_REFRESH=1 scripts/eval-ai.sh
Use scripts/agent-quality-gate.sh --mode coach --json when unsure which suite
list matches the current change.
Options:
ALCOVE_AI_EVAL_PROVIDER=claude scripts/eval-ai.sh
ALCOVE_AI_EVAL_PROVIDER=none scripts/eval-ai.sh
ALCOVE_AI_EVAL_SKIP_REFRESH=1 scripts/eval-ai.sh
Report:
- pass/fail for deterministic setup
- AI verdict and score from
.tmp/ai-eval/ai-review.json - blocking and should-fix findings first
- files changed if fixes are made
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 10d ago First seen · 45 lines · 18 tokens per session scan A 4b88b7e4373e
eval-ai is a command published in the GitHub repository OctopusGarage/alcove (0 stars, last pushed 2d ago), licensed MIT. It adds 18 tokens to every session and 344 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other commands, from other repositories
tree-ring-certify
Generate Tree Ring harness or recall-quality evidence without confusing it with the full framework release suite.
eval
Evaluate and improve one healthcare agent's system prompt. Run up to 5 iterations of: prepare fixed questions -> answer -> judge -> improve -> re-score -> commit if better.
caracterizar-prompt
Characterization de prompts/tools LLM em produção — temperature=0 + seed fixo + sanitização específica. Trata prompts como código legacy. Modernização 2026 sem precedente em 2004.
prompt-eval-debug
Debug any prompt with a tiny eval suite (control, edge, boundary), failure diagnosis, and smallest next change, no blind rewrite.
proofrag
Evaluate a RAG/LLM app — generate a golden set, judge it, and produce a scorecard.
models
Search Ryu's model catalog and optionally activate a local model.