A process for evaluating an AI or multi-agent application with a reference dataset, automated checks, scoring by a language model, and production monitoring. It records the evaluation plan in the project's documentation.

Use it to load project requirements, define or implement an evaluation suite, compare outputs with expected results, calibrate scoring, and document monitoring recommendations.

It helps turn vague expectations about AI quality into repeatable acceptance checks without inventing thresholds or risk limits that the project has not specified.

How synaptic-ai-consulting/AAMAD configures Claude Code and Cursor

Reuse

Borrowing it

Nothing to install: this file belongs to synaptic-ai-consulting/AAMAD. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.

Copy the file

curl -O https://raw.githubusercontent.com/synaptic-ai-consulting/AAMAD/main/.cursor/skills/run-evals/SKILL.md

Clone the repo

git clone --depth 1 https://github.com/synaptic-ai-consulting/AAMAD

Made for: Claude Code, Cursor.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/synaptic-ai-consulting/aamad/run-evals.svg)](https://agentmods.dev/skills/synaptic-ai-consulting/aamad/run-evals)

Your own site

<a href="https://agentmods.dev/skills/synaptic-ai-consulting/aamad/run-evals"><img src="https://agentmods.dev/badge/skills/synaptic-ai-consulting/aamad/run-evals.svg" alt="Measured on agentmods" height="20"></a>

Per session 75 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,830 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00075	$0.01830
Opus 5	$0.00037	$0.00915
Sonnet 5	$0.00015	$0.00366
Haiku 4.5	$0.00007	$0.00183

Measured yesterday against content hash fecd36e78c67, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-06, from the pricing page.

Security

Grade A, and why

run-evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured yesterday.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.cursor/skills/run-evals/SKILL.md · 111 lines

How it starts

The opening of the file, as written. The whole thing — 111 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Run Evals

Define and implement the evaluation strategy for the current AAMAD project, then write project-context/2.build/evals.md. Owned by @qa.eng; invoked as *run-evals.

For grading-ladder detail, judge calibration, a worked example, and anti-patterns to avoid, see reference.md.

Step 1: Load context

Read, in order:

project-context/1.define/prd.md, sad.md, system-description.md (if present), project-context/1.define/user-stories/*.md
project-context/2.build/backend.md, integration.md
aamad.config.yml (if present); resolve AAMAD_TARGET_RUNTIME

If sad.md section "9. Testing & Quality Assurance Specifications" already contains an evaluation criteria table (dimension, metric, threshold, grading method, source), treat it as the contract to implement — do not re-derive thresholds it already settles. Most projects built before this capability existed will not have this table; that is expected, not an error.

Step 2: Gap check — ask the operator

Business context an eval suite needs is frequently absent from project-context/. Do not invent thresholds, SLAs, or risk tolerance — per aamad-core.mdc, on missing or ambiguous inputs, write Assumptions and Open Questions rather than fabricate content.

Check for these six items. Skip any already answered by the SAD criteria table or another loaded artifact:

Accuracy threshold — what counts as a passing answer
Latency target (p95) and cost ceiling per request
Consequence of a wrong output — sets the required confidence level
Regulatory/safety constraints and any action the system must never take
Representative input distribution and where golden data comes from (real logs vs. synthetic)
Whether a human-labeled set exists for judge calibration, and which model may act as judge

If any remain unanswered, ask in a single batched round — do not interrogate one at a time. Each question offers 3–5 concrete options as usable values (not abstract labels) plus an escape hatch:

Read the full file on GitHub · 111 lines

Files

What ships with it

1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

reference.md 4.1 KB

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

yesterday First seen · 111 lines · 75 tokens per session scan A fecd36e78c67

Subscribe to this mod's changes

run-evals is a skill published in the GitHub repository synaptic-ai-consulting/AAMAD (80 stars, last pushed 2d ago), licensed Apache-2.0. It adds 75 tokens to every session and 1,830 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-05.

Building agents Testing

research-engineer

An uncompromising Academic Research Engineer. Operates with absolute scientific rigor, objective criticism, and zero flair. Focuses on theoretical correctness, formal verification, and optimal implementation across any required technology.

davila7/claude-code-templates · 43 tokens

tika-eval-compare

Compare extracts from two Tika builds over a corpus to detect regressions in content, encoding, exceptions, and embedded-document handling. Use for "compare before/after extracts", "eval this change against the corpus".

apache/tika · 50 tokens

neuron-evaluation-engineer

Create and run AI evaluations with datasets, assertions, and output drivers in Neuron AI. Use this skill whenever the user mentions evaluation, testing AI systems, creating evaluators, dataset-driven testing, assertion-based validation, or wants to measure AI system performance. Also trigger for tasks involving…

neuron-core/neuron-ai · 77 tokens

AAMAD: Skill for Claude Code