Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add simple-agent-lab/AutoTrainess --skill evalgit clone --depth 1 https://github.com/simple-agent-lab/AutoTrainessWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/simple-agent-lab/autotrainess/eval)<a href="https://agentmods.dev/skills/simple-agent-lab/autotrainess/eval"><img src="https://agentmods.dev/badge/skills/simple-agent-lab/autotrainess/eval/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/simple-agent-lab/autotrainess/eval"><img src="https://agentmods.dev/badge/skills/simple-agent-lab/autotrainess/eval.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00013 | $0.00576 |
| Opus 5.5 | $0.00005 | $0.00230 |
| Sonnet 5.5 | $0.00003 | $0.00115 |
| Haiku 4.5 | $0.00001 | $0.00058 |
Grade A, and why
eval scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 42 lines — stays where its author put it; the contents beside it link to each section on GitHub.
eval
Purpose
Run the benchmark's real evaluation on final_model/ and record reproducible evidence needed for the next stage decision.
Inputs
- Workspace repository (current working directory).
final_model/.
Required outputs
eval_results/with raw outputs or logs.- The exact evaluation command or config used.
- A concise metrics summary.
eval_results/sample_summary.mdwith 15 randomly selected evaluation samples, including score, input, target, and model output.- A brief note on the main 1-3 observed failure modes and whether each one looks more like a data problem, a training problem, or an inference/template problem.
Rules
- Use the benchmark's real evaluation entrypoint.
- If evaluation fails, stay in the benchmark's real evaluation workflow, debug the failure, and retry.
- For any evaluation used to compare checkpoints, judge model quality, or choose the next iteration, use at least
max(32, ceil(5% of the benchmark))samples. If the benchmark has fewer than 32 samples, evaluate the full benchmark. - Runs below that sample floor are allowed only as smoke tests for command or runtime validity; do not use them as evidence that one checkpoint or approach is better.
- Always produce
eval_results/sample_summary.mdwith 15 random evaluation samples. - Use
skills/eval/scripts/summarize_eval_samples.pywhen the benchmark outputs compatibleinspect_ailogs; otherwise, add the minimum benchmark-specific script or logging needed to generate the sample summary from the real evaluation run. - Keep the output focused on evidence needed for the next decision.
Procedure
- Locate the canonical evaluation entrypoint.
- If using a limited evaluation, determine the benchmark sample count and choose a limit that satisfies the sample-floor rule.
- Run evaluation on
final_model/. - Save raw outputs, commands, the sample count or limit used, and a concise metrics summary under
eval_results/. - If evaluation fails, debug it inside the benchmark's real evaluation workflow, then retry with the minimum necessary fix.
- Generate
eval_results/sample_summary.mdwith 15 random samples including score, input, target, and model output. Useskills/eval/scripts/summarize_eval_samples.pywhen compatibleinspect_ailogs are available; otherwise, add the minimum benchmark-specific script or logging needed. - Verify that
eval_results/sample_summary.mdwas generated and contains 15 samples. - Summarize the main 1-3 observed failure modes and whether each one looks more like a data problem, a training problem, or an inference/template problem.
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 42 lines · 13 tokens per session scan A 391ece948e40
eval is a skill published in the GitHub repository simple-agent-lab/AutoTrainess (22 stars, last pushed 2mo ago), licensed MIT. It adds 13 tokens to every session and 576 once invoked, about $0.0001 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
bad-case-to-dpo
A guide to turning failed production agent runs into training and evaluation data. It explains how to find the first mistake, create preference examples for DPO (Direct Preference Optimization), and check that a fix does not damage other tasks.
tool-abuse-detection
Detect tool misuse and unexpected code execution via dialogue testing. Use when the agent exposes file, code-execution, or network tools.
llm-as-judge-evaluation
Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.
meta-eval
Use when the user wants to build an evaluation system for an LLM/agent application but doesn't know where to start — they have traces, prompts, RAG pipelines, or nothing at all. Also use when the user mentions evaluation, eval, benchmarking, testing LLM quality, measuring agent performance, assessing RAG accuracy, or…
rag-eval
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG…
prompt-regression
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt…