Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/alecs5am/ralphy/evaluatornpx skills add alecs5am/ralphy --skill evaluatorgit clone --depth 1 https://github.com/alecs5am/ralphyWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/alecs5am/ralphy/evaluator)<a href="https://agentmods.dev/skills/alecs5am/ralphy/evaluator"><img src="https://agentmods.dev/badge/skills/alecs5am/ralphy/evaluator.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00216 | $0.03609 |
| Opus 5 | $0.00108 | $0.01804 |
| Sonnet 5 | $0.00043 | $0.00722 |
| Haiku 4.5 | $0.00022 | $0.00361 |
Grade A, and why
evaluator scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 162 lines — stays where its author put it; the contents beside it link to each section on GitHub.
evaluator
Trigger refinements
ALSO FIRE when the user just dropped a path that ends in .mp4 from .ralphy/workspaces/<ws>/projects/<id>/render/ with no other instructions, or when an editor handed off and the user asks "and now?" (any language).
DO NOT FIRE for unrendered projects (handback to editor for ralphy render), for raw research downloads (those go through researcher's analyze-video flow, not eval), or for source media that hasn't been composed yet.
Hard invariants
- Every model call (vision pass) routes through
cli/lib/providers/llm.ts → callLLM()via the CLI. No direct OpenAI / fal calls. - Findings are deterministic outputs of
cli/lib/eval/*— don't paraphrase them; pass through verbatim to the fixer agent. - Keyframe slicing is a cheap diagnostic, NOT a ship gate (#411). A keyframe-only (or structure-only) report can NEVER mark a Unit ship-ready —
report.gate.shipReadyis hard-false on it. The final gate before forming/publishing a Unit is the native-video pass (full mp4 → model), or deep-style when a STYLE_LOCK/brief exists. Screenshot slicing misses temporal continuity, audio-picture alignment, pacing, and caption sync — exactly the failures that hallucinate when the model only sees stills.
Where this sits in the Unit lifecycle. Eval is phase 14 of the canonical Unit lifecycle. Its native-video gate is the one that flips
polishedtotrueinralphy project status <id> --contract; thequality-gate-failedandnative-gate-requiredstop conditions are derived fromeval.json'sgate/scoring.verdict. Ablockverdict feeds the repair loop (phase 15) and the optional polish council (phase 16).
You evaluate rendered UGC videos and produce a report that another agent (the fixer) can act on without reading the video itself. The contract is: the report is the handoff.
What this skill is not
- Not a researcher tool. For "analyze this TikTok/Reel from a creator I want to imitate", route to
/researcher. - Not a fixer. The findings list is meant to be read by a separate agent (or the editor / art-director / scenarist) that will execute the fixes. Don't try to fix issues from inside this skill — that's a different role and would skip the user's chance to triage.
- Not a publisher / scheduler. Verdict is informational, not a publish gate.
What ships with it
1 file beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 162 lines · 216 tokens per session scan A 5b6eadead27e
evaluator is a skill published in the GitHub repository alecs5am/ralphy (129 stars, last pushed 9d ago), licensed Apache-2.0. It adds 216 tokens to every session and 3,609 once invoked, about $0.0011 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
audio-mixing-mastering
Provider-independent audio mixing and mastering direction for AI agents finishing generated videos, ads, trailers, explainers, podcasts, recuts, avatar clips, music videos, documentaries, and social content. Use when planning, mixing, repairing, mastering, QCing, or delivering dialogue, music, ambience, and sound…
captions-media-accessibility
Provider-independent captions and media accessibility direction for AI agents producing or finishing generated videos, ads, social clips, explainers, avatar videos, documentaries, podcasts/video recuts, training media, and localized content. Use when planning, authoring, reviewing, localizing, burning in, exporting…
avatar-spokesperson-production
Provider-independent production workflow for AI avatar spokesperson videos, including presenter briefs, consent and likeness rights, disclosure and platform labeling, casting, localization, performance direction, brand fit, claims review, lip-sync/audio QA, accessibility, approvals, and delivery packages. Use when…
music-video-production
Provider-independent production workflow for AI agents creating music videos, lyric videos, visualizers, performance/narrative/dance videos, social cutdowns, album or single launch assets, and generated-media clips synchronized to songs. Use when planning, directing, prompting, editing, reviewing, or delivering video…
podcast-production
Produce audio-first podcast episodes with generative tools — design the show and episode format (interview, narrative, news brief, two-host conversational), write scripts for the ear, decide between fully-synthetic and hybrid (recorded human + synthetic) production and meet the disclosure duty each triggers, cast and…
talking-head-podcast-recut
Provider-independent production workflow for recutting long talking-head, podcast, interview, webinar, panel, lecture, livestream, founder call, or customer-call footage into short clips, highlight reels, trailers, audiograms, captioned vertical videos, and social cutdowns. Use when an agent must ingest source…