Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx skills add nearform/lastlight --skill lastlight-evals-loopgit clone --depth 1 https://github.com/nearform/lastlightWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/nearform/lastlight/lastlight-evals-loop)<a href="https://agentmods.dev/skills/nearform/lastlight/lastlight-evals-loop"><img src="https://agentmods.dev/badge/skills/nearform/lastlight/lastlight-evals-loop/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/nearform/lastlight/lastlight-evals-loop"><img src="https://agentmods.dev/badge/skills/nearform/lastlight/lastlight-evals-loop.svg" alt="Reviewed on agentmods" width="80" height="20"></a>- NVIDIA SkillSpector warn
SkillSpector: 3 findings, up to medium
These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →
- medium MCP Rug Pull · line 86 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
- medium MCP Rug Pull · line 161 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
- medium MCP Rug Pull · line 171 npx commands without a version suffix (e.g. @1.0.0) create a rug-pull risk if the upstream server is compromised and publishes a malicious update.Fix: Pin the version: npx @scope/[email protected]
What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00200 | $0.02785 |
| Opus 5.5 | $0.00080 | $0.01114 |
| Sonnet 5.5 | $0.00040 | $0.00557 |
| Haiku 4.5 | $0.00020 | $0.00279 |
Grade A, and why
lastlight-evals-loop scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 211 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Improve an eval, honestly
A loop that raises an eval score without gaming it. The trap: the loop can see both the gold answer and the agent's answer for every case, so it could overfit — hardcode a repo's finding into a prompt, or edit a gold answer to force a pass. That lifts the number and ruins the reviewer. Every guardrail here exists to force generic, cross-cutting changes that would help on repos the loop has never seen.
Scope: the pr-review tier (its judge trace gives you the agent-vs-gold detail diagnosis needs). The pattern extends to triage/code-fix, but start here.
The method — weakness-mine → propose a few minimal candidates → keep the one that survives a blind held-out gate — follows Self-Harness: Harnesses That Improve Themselves (arXiv:2606.09498), adapted to keep our anti-gaming discipline. See references/approach.md for the mapping.
Prerequisites
- A scaffolded evals workspace with a pr-review dataset and a working provider
key — set up via
lastlight-evalsfirst if you don't have one. Confirm a barelastlight-evals run pr-review --limit 1 --no-opengrades a case. - An
instance/overlay (the deployment overlay) to receive generic edits — seelastlight-overlay. The loop edits copies here, never core. - Two helpers ship with the evals package:
scripts/mine-failures.ts(ranks the TRAIN failure signatures into an evidence bundle — the diagnosis input) andscripts/diff-runs.ts(compares two runs and calls keep/revert).
The one rule
One change kept per round. A round may explore a few minimal candidates,
but at most one is ever kept and committed to instance/ — the rest are reverted.
So which edit moved the number is always attributable, and no overfit edit rides
along with a good one. Never keep two edits from one round without re-measuring
each in isolation.
Start here
| You have… | Go to |
|---|---|
| Never run this loop on this dataset | §1 Set up the split then §2 |
| A split + baseline already recorded | §3 Diagnose (mine failures) |
| Candidate change(s) ready to test | §5 Audit → §6 Apply → §7 Measure → §8 Decide |
What ships with it
4 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 211 lines · 200 tokens per session scan A c09da0484307
lastlight-evals-loop is a skill published in the GitHub repository nearform/lastlight (39 stars, last pushed yesterday), licensed MIT. It adds 200 tokens to every session and 2,785 once invoked, about $0.0008 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
benchmark-design
Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline, or import a published Benchmark package.
agent-evaluation
Run one specified Test Agent on one specified Benchmark Case exactly once, privately score that execution, and return one protocol result.
agent-optimization
Improve an Agent State through versioned scores and score-linked Traces from a frozen Benchmark.
analyze-task
Check OSWorld tasks. Validate the evaluation function, verify that the instruction is feasible given the task setup and agent-visible files, inspect setup artifacts when needed, and produce both markdown and structured JSON reports.
review
Review a small code change for correctness and missing tests.
orca
Use when the user asks to run an Orca flow (e.g. "use orca to implement …"), or when delegating a well-defined implementation task to Orca's autonomous multi-agent flow — a headless plan-code-review CLI to hand a coding task to instead of implementing it yourself.