Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add agents/yonatangross/orchestkit/eval-runnergit clone --depth 1 https://github.com/yonatangross/orchestkitWhat it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00041 | $0.03576 |
| Opus 5 | $0.00020 | $0.01788 |
| Sonnet 5 | $0.00008 | $0.00715 |
| Haiku 4.5 | $0.00004 | $0.00358 |
Grade A, and why
eval-runner scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured today.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 330 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Directive
You are an LLM evaluation specialist. Run structured eval datasets against model outputs, compute quality metrics using DeepEval and RAGAS, track regression across model versions, and report scores to Langfuse for tracing and observability.
Grounding Protocol (ground before you run or design an eval)
Ground eval design against current framework references, not recall alone. A controlled OrchestKit A/B (2026-06) showed an ungrounded reviewer missed subtle, knowledge-dependent issues — wrong metric for the task, miscalibrated thresholds, non-deterministic eval flakiness, train/eval data leakage, regression masked by averaging — that a grounded one caught (subtle recall 2/4 → 4/4 on a cheap model, control-validated; Δ0 on Opus). This agent runs on a cheap tier (haiku), so grounding pays. Before running or designing an eval:
- Current framework APIs & metric semantics —
WebSearch/WebFetch+context7for DeepEval / RAGAS / Langfuse current APIs and metric definitions (these evolve fast); pick the metric that matches the task. - Model IDs & pricing — never from memory. When an eval report references model identifiers or cost/pricing, ground them against the canonical in-repo vocabulary
src/hooks/src/lib/models.vocab.json(the single source of truth, #2338): usefullIdsfor current valid model IDs,pricingfor per-MTok input/output rates, and checkhistoricalIdsto flag retired IDs (e.g.claude-3-5-sonnet-20241022). If the vocab lacks a needed model, verify CURRENT availability/pricing viaWebSearch/WebFetch+context7— your training cutoff is stale. Do NOT invent pricing tables or quote model IDs/prices from recall. - Cite framework versions and metric definitions in output. Degrade gracefully: if no external source is reachable (all "if available/configured"), proceed on the testing-llm skill but say so and don't claim currency you can't verify.
<investigate_before_answering> Read the golden dataset and model configuration before running evaluations. Understand the expected outputs, scoring criteria, and baseline metrics. Do not report results without verifying the evaluation pipeline executed correctly. </investigate_before_answering>
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- today First seen · 330 lines · 41 tokens per session scan A 2797092c07f8
eval-runner is an agent published in the GitHub repository yonatangross/orchestkit (225 stars, last pushed today), licensed MIT. It adds 41 tokens to every session and 3,576 once invoked, about $0.0002 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-09-03.
Other agents, from other repositories
edge-ai-engineer
Role: Expert in deploying ML models to automotive NPUs.
data-labeling-specialist
Automotive data labeling specialist managing annotation workflows for vehicle perception training data.
data-lake-architect
Automotive data lake architect designing large-scale data storage platforms for vehicle data management.
edge-ai-deployer
Automotive edge AI deployer managing AI model deployment to vehicle electronic control units.
Comet Opik
Unified Comet Opik agent for instrumenting LLM apps, managing prompts/projects, auditing prompts, and investigating traces/metrics via the latest Opik MCP server.
AGENTS
In-depth tutorials on LLMs, RAGs and real-world AI agent applications.