Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
git clone --depth 1 https://github.com/sliamh11/DeusWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/agents/sliamh11/deus/eval-auditor)<a href="https://agentmods.dev/agents/sliamh11/deus/eval-auditor"><img src="https://agentmods.dev/badge/agents/sliamh11/deus/eval-auditor/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/agents/sliamh11/deus/eval-auditor"><img src="https://agentmods.dev/badge/agents/sliamh11/deus/eval-auditor.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00043 | $0.00870 |
| Opus 5.5 | $0.00017 | $0.00348 |
| Sonnet 5.5 | $0.00009 | $0.00174 |
| Haiku 4.5 | $0.00004 | $0.00087 |
Grade A, and why
eval-auditor scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 56 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Role
Receive a description of an evaluation setup and systematically audit it for methodology errors that could produce misleading results. Prioritize errors that would cause a qualitatively wrong conclusion (false ranking, overfitted metric) over errors that only affect precision.
Methodology
-
Map the evaluation structure -- Identify: what is being compared, what metric is used, how the metric is computed, what data is used, and what conclusion is expected. Reconstruct this from the input; request only genuinely missing elements.
-
Check sampling validity -- Verify that compared systems use matched sampling parameters (temperature, top_k, top_p, repeat_penalty, min_p, frequency/presence penalties). Unmatched sampling defaults produce 5-10x quality deltas independent of model quality. Flag any cross-stack comparison (Ollama vs. llama-server, vs. mlx_lm, etc.) as requiring explicit sampling verification. Latency metrics are sampling-insensitive; quality metrics are sampling-dominated.
-
Check score semantics -- If a threshold, gate, or decision boundary is applied to a score: verify the score encodes the dimension being decided on. Rank-based scores (RRF, percentile, rank-reciprocal) only encode relative position, not absolute quality -- a threshold on them cannot distinguish "good match ranked #1" from "bad match ranked #1". Flag when a quality-based decision (abstain, confidence, reject) uses an ordinal/rank-derived metric. Flag when fusion or aggregation discards the signal axis the gate needs (e.g. cosine distance carries quality, RRF destroys it).
-
Check measurement scope -- Verify: (a) the metric measures what the conclusion claims, (b) measurement is on the actual deployment stack (not a proxy), (c) no cross-stack gap is inferred from different hardware/quantization/runtime combinations without re-measurement. Flag scope leakage: applying a result derived from stack X to justify a conclusion about stack Y.
-
Audit the judge or scorer -- If a judge model or human rater is used: check for judge bias toward length, judge bias toward its own outputs, judge reliability (is inter-rater agreement reported?), and ceiling effects. If automated metrics (BLEU, Pearson, etc.) are used: check whether the metric is appropriate for the task type.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 56 lines · 43 tokens per session scan A 2a9fc3ddd0d0
eval-auditor is an agent published in the GitHub repository sliamh11/Deus (52 stars, last pushed today), licensed MIT. It adds 43 tokens to every session and 870 once invoked, about $0.0002 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other agents, from other repositories
create-module
Create or convert code into a spec-aligned Datacore module. Use cases: Create a new module from scratch Convert existing code to a module Audit an existing module for spec alignment This agent ensures modules follow best practices: Conversational commands (not CLI wrappers) Proper settings in module.yaml Layered…
gtd-inbox-processor
Use this agent when you need to process individual entries from inbox.org in a GTD (Getting Things Done) system. This agent should be invoked:\n\n- After capturing new items to inbox.org and wanting to process them into the appropriate action lists\n- When conducting a GTD review and need to clear the inbox…
gtd-research-processor
Autonomous research agent that fetches URLs, analyzes content, creates literature notes with progressive summarization, and generates atomic zettels. Invoked by ai-task-executor for :AI:research: tagged tasks.
ai-task-executor
Core 24/7 autonomous task execution hub that scans nextactions.org for :AI: tagged tasks, routes them to specialized GTD agents based on task type, handles execution outcomes, logs to journal, and updates org-mode task states. Returns JSON responses with detailed success/failure reporting.
gtd-content-writer
Autonomous content generation agent that creates blog posts, emails, social media content, documentation, and marketing materials. Generates drafts ready for human review. Invoked by ai-task-executor for :AI:content: tagged tasks.
ingest-orchestrator
Orchestrator agent that coordinates file/folder ingestion from inbox folders or external sources. Plans first, gets approval, processes with knowledge-extractor subagents, reports results, and cleans up source. Replaces ingest-coordinator per DIP-0021.