A framework for testing whether an AI agent produces acceptable results. Each test gives the agent an input and checks its answer with either fixed code rules or another language model acting as a judge.

Use it to define evaluation cases, run capability and regression test suites, and measure reliable success across repeated trials.

It turns vague impressions about agent quality into repeatable tests. Running cases multiple times also shows whether an agent is consistently correct or only succeeds occasionally.

Evals — Skill by danielmiessler

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods

npx agentmods add skills/danielmiessler/lifeos/evals

Any agent

npx skills add danielmiessler/LifeOS --skill evals

Clone the repo

git clone --depth 1 https://github.com/danielmiessler/LifeOS

Made for: Claude Code, Codex.

Per session 129 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 2,181 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. Scan, not verified.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5	$0.00129	$0.02181
Opus 5	$0.00064	$0.01091
Sonnet 5	$0.00026	$0.00436
Haiku 4.5	$0.00013	$0.00218

Measured 3d ago against content hash 0d317edd544c, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

Evals scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

The scan reads SKILL.md. This mod also ships 28 executable files (Graders/Base.ts, Graders/CodeBased/BinaryTests.ts, Graders/CodeBased/index.ts, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

LifeOS/install/skills/Evals/SKILL.md · 114 lines

How it starts

The opening of the file, as written. The whole thing — 114 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Evals — Assertion-First AI Evaluation

What it is

An eval gives an AI an input, then applies assertions to its output to measure success (Anthropic's definition). A case is {id, prompt, assert:[...]}. Each assertion is either deterministic (code, fast/free) or model-graded (an LLM judge). Cases run multiple trials; we report pass^k (all trials pass — the honest metric for a reliability-critical agent) and pass@k (any trial passes). Everything routes through Inference.ts — subscription-billed, no API-key path, no external deps.

Grounded in Anthropic's current doctrine — Demystifying evals for AI agents, Define success criteria / develop tests, and the skill-creator {text, passed, evidence} assertion convention. The typed-assert layer is promptfoo-shaped but our own TS.

Freshness contract: "aligned to Anthropic's doctrine" is a live claim, not a snapshot. When designing a new suite class or touching the ## Doctrine section below, re-fetch the Demystifying-evals doc and flag where it has moved past what's encoded here. Advisory only — report divergence, never auto-adopt, and an unreachable URL never blocks a run.

The canonical path (v2)

Tool	Role
`Tools/Assertions.ts`	Deterministic assert engine: `equals`, `contains`, `icontains`, `contains-all/any`, `regex`, `starts-with`, `ends-with`, `is-json`, `contains-json`, `max-length`, `min-length`, each with `not-` negation. Sync, no model call.
`Tools/Judge.ts`	Model-graded asserts `llm-rubric` (1–5 → 0–1, threshold) and `llm-assert` (NL assertions → TRUE/FALSE/UNKNOWN). Forced-structured JSON verdict, reason-then-score, distinct judge level, Unknown→miss escape hatch.
`Tools/EvalRunner.ts`	Loads a suite, runs the agent-under-test per case (single-shot inference against the target system prompt), applies asserts, computes pass^k/pass@k, persists transcripts + `latest.json`.
`Tools/SuiteManager.ts`	Suite listing + saturation tracking.
`Tools/FailureToTask.ts`	Convert real failures into cases (seed from 20–50 real failures).

Read the full file on GitHub · 114 lines

Files

What ships with it

52 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

3d ago First seen · 114 lines · 129 tokens per session scan A 0d317edd544c

Subscribe to this mod's changes

Evals is a skill published in the GitHub repository danielmiessler/LifeOS (18,798 stars, last pushed 19d ago), licensed MIT. It adds 129 tokens to every session and 2,181 once invoked, about $0.0006 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Building agents Testing ai ai-harness augmentation humans intent-engineering

mobile-harness

Portable Android and iOS device-control harness for agents. Use when an agent needs to control or automate Android/iOS devices — tap, swipe, screenshot, app control — locally (ADB/Portal/Simulator) or on Mobilerun cloud devices.

droidrun/mobile-harness · 54 tokens

macos-harness

Control a whole Mac from one persistent Python session with screenshots, PID-targeted input, an animated virtual pointer, targeted Apple Accessibility, Apple Events, Browser Harness CDP, and filesystem access. Use for native, Electron, browser, dialog, file, or cross-app tasks without moving the physical cursor or…

Zfinix/aster · 71 tokens

aster-config

Reference for aster.yaml, covering review models, analyzers, focus areas, include/exclude globs, minconfidence, and the permissions block that gates edits. Use when creating or editing aster.yaml, choosing hypothesis/verify models, enabling semgrep or ast-grep, or configuring edit permissions.

Zfinix/aster · 65 tokens

aster-chat-sessions

Drive aster chat programmatically and manage sessions and memory: one-shot --print/--json answers, --messages-json for caller-owned history, --continue and --session persistence, --allow-edits gating, and the sessions/memory commands. Use when scripting aster chat, integrating it into an editor or agent, or inspecting…

Zfinix/aster · 74 tokens

aster-review-ci

Run aster code reviews non-interactively in CI, GitHub Actions, or from another agent. Covers aster review --pr, --json, --stream, --comment, diff-from-stdin, token handling, and filtering findings. Use when wiring aster into a pipeline, posting PR comments, or parsing review output programmatically.

Zfinix/aster · 75 tokens

echo-plugins

Build, validate, package, and stage optional Echo plugins using the Echo Plugin API v1 security and lifecycle contracts.

BrentFarris/echo · 27 tokens