LLM Evaluation System is an agent-guided platform for evaluating language models and agents, generating datasets and configuring multiple judges from natural-language requests before producing an analysis report. It is for comparing model responses, testing agents, and creating document-grounded evaluation data.
Borrowing it
Nothing to install: this file belongs to awslabs/llm-evaluation-system. Take a copy, put it at the same path in your own repository, and replace the rules that are about this project with yours.
curl -O https://raw.githubusercontent.com/awslabs/llm-evaluation-system/main/.claude/skills/ship-it/SKILL.mdgit clone --depth 1 https://github.com/awslabs/llm-evaluation-systemWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/awslabs/llm-evaluation-system/ship-it)<a href="https://agentmods.dev/skills/awslabs/llm-evaluation-system/ship-it"><img src="https://agentmods.dev/badge/skills/awslabs/llm-evaluation-system/ship-it.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00131 | $0.02504 |
| Opus 5 | $0.00066 | $0.01252 |
| Sonnet 5 | $0.00026 | $0.00501 |
| Haiku 4.5 | $0.00013 | $0.00250 |
Grade C, and why
ship-it scanned grade C with 2 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Downloads and executes remote codehighSupply chain
curl | sh runs whatever the server returns today, which is not necessarily what it returned when this was reviewed.
| Verify PyPI | `curl -s https://pypi.org/pypi/llm-evaluation-system/json \| python3 -c "import sys,json; print(json.load(sys.stdin)['info']['version'])"` | Makes network callslowCapability
Not a fault in itself. Listed so you know the mod talks to something, and to what.
curl -s https://pypi.org/pypi/llm-evaluation-system/json | \ How it starts
The opening of the file, as written. The whole thing — 252 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Ship It
This skill ships changes in the llm-evaluation-system repo from "I have
local changes" through to "merged + (optionally) on PyPI." It exists
because this repo has specific conventions that are easy to get wrong
individually and straightforward when followed as a unit.
Read the whole skill before acting if it's your first invocation in a session. After that, the quick reference at the bottom is usually enough.
Conventions this repo follows
- Conventional Commits for every commit message AND every PR title.
Valid prefixes:
feat,fix,docs,chore,refactor,test,build,ci,perf,style,revert. Usefeat(scope):when a scope adds clarity (e.g.feat(mcp): ...,fix(release): ...). - Never push directly to
main. Every change goes through a PR. Direct pushes are blocked by the auto-mode classifier anyway, but the deeper reason is reviewability and not bypassing CI checks onpublish.ymlpaths. - Never force-push. Rewriting public history breaks others' clones and is essentially never the right answer.
- Releases are deliberate human actions. Use
make release/make release-minor/make release-majorfrom a cleanmain. There is no auto-release on merge (this was explored and rejected; see the Releasing section below for why). - After PRs merge, branches need manual cleanup (auto-delete head branches isn't enabled on this repo).
Branch state
You should already be on a feature branch with your changes — worktree-based by default, per CLAUDE.md. This skill picks up from there.
If you're invoked from main with uncommitted changes, stop and ask
the user before doing anything. Moving in-flight work into a worktree
retroactively is awkward (stash + apply, or branch in place, or
something else), and the user should pick which.
Workflow: commit and open a PR
- Verify the working tree state. Run
git status --short. If there are untracked or modified files that aren't part of THIS change (e.g. someone else's experimental directory, build artifacts from a previous run, leftover test files), pause and check with the user before touching them. Don't auto-stage withgit add -Aorgit add ..
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 8d ago First seen · 252 lines · 131 tokens per session scan C f3819e416e1a
ship-it is a skill published in the GitHub repository awslabs/llm-evaluation-system (23 stars, last pushed 5d ago), licensed Apache-2.0. It adds 131 tokens to every session and 2,504 once invoked, about $0.0007 per session on Opus 5. A static security scan graded it C with 2 findings (downloads and executes remote code, makes network calls). No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
github-automation
GitHub workflow automation, PR management, issue tracking, and code review coordination. Integrates with GitHub Actions and repository management. Use when: PR creation, code review, issue management, release automation, workflow setup. Skip when: local-only changes, non-GitHub repositories.
comet-safe-delivery
A Chinese-language procedure for safely delivering specified Comet changes through Git. It covers checking worktrees and unrelated edits, staging exact files, validating hooks, and authorized commits or pushes.
changelog
Auto-generates a changelog from git commits, sprint data, and design documents. Produces both internal and player-facing versions.
close-task-commit-push-pr
Close the active backlog task (detected from branch name), commit all changes, push to remote, and open a pull request. Use when the user says "close task and ship it", "close task commit push pr", or invokes /close-task-commit-push-pr.
git-workflow
Guides you through Git workflows — branching strategies, commit conventions, merge conflict resolution, and release management. Use when working with Git repositories or when the user asks about version control best practices.
development-workflow
Detailed development workflow with modular patterns for git, review, testing, and deployment.