LLM Evaluation System is an agent-guided platform for evaluating language models and agents, generating datasets and configuring multiple judges from natural-language requests before producing an analysis report. It is for comparing model responses, testing agents, and creating document-grounded evaluation data.
Borrowing it
Nothing to install: this file belongs to awslabs/llm-evaluation-system. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/awslabs/llm-evaluation-system/main/.claude/skills/ship-it/SKILL.mdgit clone --depth 1 https://github.com/awslabs/llm-evaluation-systemWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/awslabs/llm-evaluation-system/ship-it)<a href="https://agentmods.dev/skills/awslabs/llm-evaluation-system/ship-it"><img src="https://agentmods.dev/badge/skills/awslabs/llm-evaluation-system/ship-it/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/awslabs/llm-evaluation-system/ship-it"><img src="https://agentmods.dev/badge/skills/awslabs/llm-evaluation-system/ship-it.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00131 | $0.02504 |
| Opus 5.5 | $0.00052 | $0.01002 |
| Sonnet 5.5 | $0.00026 | $0.00501 |
| Haiku 4.5 | $0.00013 | $0.00250 |
Grade C, and why
ship-it scanned grade C with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Downloads and executes remote codehighSupply chain
curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.
| Verify PyPI | `curl -s https://pypi.org/pypi/llm-evaluation-system/json \| python3 -c "import sys,json; print(json.load(sys.stdin)['info']['version'])"` | Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
curl -s https://pypi.org/pypi/llm-evaluation-system/json | \ How it starts
The opening of the file, as written. The whole thing — 252 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Ship It
This skill ships changes in the llm-evaluation-system repo from "I have
local changes" through to "merged + (optionally) on PyPI." It exists
because this repo has specific conventions that are easy to get wrong
individually and straightforward when followed as a unit.
Read the whole skill before acting if it's your first invocation in a session. After that, the quick reference at the bottom is usually enough.
Conventions this repo follows
- Conventional Commits for every commit message AND every PR title.
Valid prefixes:
feat,fix,docs,chore,refactor,test,build,ci,perf,style,revert. Usefeat(scope):when a scope adds clarity (e.g.feat(mcp): ...,fix(release): ...). - Never push directly to
main. Every change goes through a PR. Direct pushes are blocked by the auto-mode classifier anyway, but the deeper reason is reviewability and not bypassing CI checks onpublish.ymlpaths. - Never force-push. Rewriting public history breaks others' clones and is essentially never the right answer.
- Releases are deliberate human actions. Use
make release/make release-minor/make release-majorfrom a cleanmain. There is no auto-release on merge (this was explored and rejected; see the Releasing section below for why). - After PRs merge, branches need manual cleanup (auto-delete head branches isn't enabled on this repo).
Branch state
You should already be on a feature branch with your changes — worktree-based by default, per CLAUDE.md. This skill picks up from there.
If you're invoked from main with uncommitted changes, stop and ask
the user before doing anything. Moving in-flight work into a worktree
retroactively is awkward (stash + apply, or branch in place, or
something else), and the user should pick which.
Workflow: commit and open a PR
- Verify the working tree state. Run
git status --short. If there are untracked or modified files that aren't part of THIS change (e.g. someone else's experimental directory, build artifacts from a previous run, leftover test files), pause and check with the user before touching them. Don't auto-stage withgit add -Aorgit add ..
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 2mo ago First seen · 252 lines · 131 tokens per session scan C f3819e416e1a
ship-it is a skill published in the GitHub repository awslabs/llm-evaluation-system (26 stars, last pushed yesterday), licensed Apache-2.0. It adds 131 tokens to every session and 2,504 once invoked, about $0.0005 per session on Opus 5.5. A static security scan graded it C with 2 findings (downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
ship-it compared with skills from other repositories
| Mod | Type | Stars | Last push | Licence | Scan | Tokens | Works with |
|---|---|---|---|---|---|---|---|
| ship-itawslabs/llm-evaluation-system | Skill | 26 | yesterday | Apache-2.0 | C | 131 | Claude Code |
| github-automationruvnet/ruflo | Skill | 81k | yesterday | MIT | A | 61 | Claude Code |
| changelogDonchitos/Claude-Code-Game-Studios | Skill | 36k | 19d ago | MIT | A | 22 | Claude Code |
| comet-safe-deliveryrpamis/comet | Skill | 3.3k | yesterday | MIT | A | 78 | Claude Code, Codex |
| coco-shipcoco-research/coco | Skill | 785 | yesterday | Apache-2.0 | A | 69 | Claude Code |
| close-task-commit-push-prdevoxx/DevoxxGenieIDEAPlugin | Skill | 687 | yesterday | MIT | A | 64 | Claude Code |
| git-workflowjezweb/claude-skills | Skill | 1.1k | 28d ago | MIT | A | 52 | Claude Code |