Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/nearform/lastlight/lastlight-evals-loopnpx skills add nearform/lastlight --skill lastlight-evals-loopgit clone --depth 1 https://github.com/nearform/lastlightWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/nearform/lastlight/lastlight-evals-loop)<a href="https://agentmods.dev/skills/nearform/lastlight/lastlight-evals-loop"><img src="https://agentmods.dev/badge/skills/nearform/lastlight/lastlight-evals-loop.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00200 | $0.02785 |
| Opus 5 | $0.00100 | $0.01392 |
| Sonnet 5 | $0.00040 | $0.00557 |
| Haiku 4.5 | $0.00020 | $0.00279 |
Grade A, and why
lastlight-evals-loop scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 6d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 211 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Improve an eval, honestly
A loop that raises an eval score without gaming it. The trap: the loop can see both the gold answer and the agent's answer for every case, so it could overfit — hardcode a repo's finding into a prompt, or edit a gold answer to force a pass. That lifts the number and ruins the reviewer. Every guardrail here exists to force generic, cross-cutting changes that would help on repos the loop has never seen.
Scope: the pr-review tier (its judge trace gives you the agent-vs-gold detail diagnosis needs). The pattern extends to triage/code-fix, but start here.
The method — weakness-mine → propose a few minimal candidates → keep the one that survives a blind held-out gate — follows Self-Harness: Harnesses That Improve Themselves (arXiv:2606.09498), adapted to keep our anti-gaming discipline. See references/approach.md for the mapping.
Prerequisites
- A scaffolded evals workspace with a pr-review dataset and a working provider
key — set up via
lastlight-evalsfirst if you don't have one. Confirm a barelastlight-evals run pr-review --limit 1 --no-opengrades a case. - An
instance/overlay (the deployment overlay) to receive generic edits — seelastlight-overlay. The loop edits copies here, never core. - Two helpers ship with the evals package:
scripts/mine-failures.ts(ranks the TRAIN failure signatures into an evidence bundle — the diagnosis input) andscripts/diff-runs.ts(compares two runs and calls keep/revert).
The one rule
One change kept per round. A round may explore a few minimal candidates,
but at most one is ever kept and committed to instance/ — the rest are reverted.
So which edit moved the number is always attributable, and no overfit edit rides
along with a good one. Never keep two edits from one round without re-measuring
each in isolation.
Start here
| You have… | Go to |
|---|---|
| Never run this loop on this dataset | §1 Set up the split then §2 |
| A split + baseline already recorded | §3 Diagnose (mine failures) |
| Candidate change(s) ready to test | §5 Audit → §6 Apply → §7 Measure → §8 Decide |
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 6d ago First seen · 211 lines · 200 tokens per session scan A c09da0484307
lastlight-evals-loop is a skill published in the GitHub repository nearform/lastlight (22 stars, last pushed yesterday), licensed MIT. It adds 200 tokens to every session and 2,785 once invoked, about $0.0010 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
agent-loop
Production Claude agent loop — Session/Harness/Registry/Tool abstraction, DRYRUN safety guard, APScheduler integration, dead-letter error handling, tool registry, and observability hooks for autonomous agent systems.
example-skill
Counts lines, words, and bytes in a text file using the bundled count.sh script. Use when asked to count/measure a file's size in these terms.
design-taste-frontend
Anti-slop frontend skill for landing pages, portfolios, and redesigns. The agent reads the brief, infers the right design direction, and ships interfaces that do not look templated. Real design systems when applicable, audit-first on redesigns, strict pre-flight check.
open-code-review
Performs AI-powered code review on Git changes using the ocr CLI from alibaba/open-code-review. Use when the user asks to review code, review a pull request, review staged/unstaged changes, review a commit, or compare branches for code quality issues. Produces line-level review comments and can automatically apply…
image-to-code
Elite website image-to-code skill for Codex. For visually important web tasks, it must first generate the design image(s) itself, deeply analyze them, then implement the website to match them as closely as possible. In Codex, it must prefer large, readable, section-specific images instead of tiny compressed boards…
brandkit
Premium brand-kit image generation skill for creating high-end brand-guidelines boards, logo systems, identity decks, and visual-world presentations. Trained for minimalist, cinematic, editorial, dark-tech, luxury, cultural, security, gaming, developer-tool, and consumer-app brand systems. Optimized for intentional…