Getting it into your agent
It runs from inside its repository, so the clone comes first — what it calls does not travel with the file alone.
git clone --depth 1 https://github.com/stefan-stepzero/shipkitnpx agentmods add skills/stefan-stepzero/shipkit/shipkit-semantic-qaWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/stefan-stepzero/shipkit/shipkit-semantic-qa)<a href="https://agentmods.dev/skills/stefan-stepzero/shipkit/shipkit-semantic-qa"><img src="https://agentmods.dev/badge/skills/stefan-stepzero/shipkit/shipkit-semantic-qa/github.svg" alt="Measured on agentmods" height="20"></a>Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.
<a href="https://agentmods.dev/skills/stefan-stepzero/shipkit/shipkit-semantic-qa"><img src="https://agentmods.dev/badge/skills/stefan-stepzero/shipkit/shipkit-semantic-qa.svg" alt="Reviewed on agentmods" width="80" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5.1 | $0.00078 | $0.07344 |
| Opus 5.5 | $0.00031 | $0.02938 |
| Sonnet 5.5 | $0.00016 | $0.01469 |
| Haiku 4.5 | $0.00008 | $0.00734 |
Grade A, and why
shipkit-semantic-qa scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 1mo ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 672 lines — stays where its author put it; the contents beside it link to each section on GitHub.
shipkit-semantic-qa - Semantic Quality Assurance
Purpose: Define test inputs and quality criteria, generate test scripts, run them, and let Claude semantically judge outputs (API responses or UI screenshots) against human-defined criteria.
Pattern: One skill, one loop — Setup → Run → Judge. Two suite types: backend (API/LLM pipeline) and frontend (visual components).
When to Invoke
User triggers:
- "Semantic QA", "Set up QA", "Quality check"
- "Visual QA", "Screenshot QA", "Check my UI"
- "Judge outputs", "Run QA suite", "Check quality"
- "Set up quality criteria", "Define test inputs"
Workflow position:
- After features are implemented (something to test)
- Before verify/preflight (catches quality issues early)
- Can run standalone against any API or UI
Prerequisites
Required: None (Setup mode creates everything)
Helpful:
.shipkit/stack.json— Tech stack informs script generation.shipkit/specs/— Acceptance criteria can seed quality criteria- Playwright installed (for frontend suites only)
Process
Completion Tracking
In --full mode (all 3 phases sequential), create tasks at the start:
TaskCreate: "Setup: Define criteria + generate test script"TaskCreate: "Run: Execute tests + verify output count"TaskCreate: "Judge: Evaluate ALL outputs against ALL criteria"TaskCreate: "Write judgment.md + judgment.json"
TaskUpdate each task to in_progress when starting it, completed when done.
In Judge mode with 5+ outputs, create one task per output to prevent partial evaluation. Do NOT present judgment summary until all output evaluations are complete and files are written.
Step 0: Mode Detection
Determine mode from arguments and state:
Explicit flag provided:
--setup→ Setup mode--run→ Run mode--judge→ Judge mode--full→ Setup (if needed) → Run → Judge--fidelity→ Fidelity mode — score a built app against captured intent (completeness + essence), emit a fidelity scorecard. See Step 5. Add a second suite/arm to compare two builds under one rubric.
What ships with it
19 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.
- references/criteria-guide.md 4.1 KB
- references/example.json 4.8 KB
- references/fidelity-scorecard-schema.md 24 KB
- references/output-schema.md 2.9 KB
- references/README.md 757 B
- tools/fidelity/_common.py 2.5 KB runs code
- tools/fidelity/_declared.py 15 KB runs code
- tools/fidelity/_smoke/coach/leaderboard.py 293 B runs code
- tools/fidelity/_smoke/ExecView.tsx 298 B
- tools/fidelity/_smoke/pages/index.tsx 311 B
- tools/fidelity/_smoke/PredictivePacing.tsx 600 B
- tools/fidelity/_smoke/scratch/ExperimentPanel.tsx 724 B
- tools/fidelity/_smoke/spec.json 2.9 KB
- tools/fidelity/_smoke/StaffDashboard.tsx 466 B
- tools/fidelity/fidelity-score.py 30 KB runs code
- tools/fidelity/mock-seam-detector.py 11 KB runs code
- tools/fidelity/README.md 8.6 KB
- tools/fidelity/ssot-checker.py 6.8 KB runs code
- tools/fidelity/unbacked-surface-checker.py 6.3 KB runs code
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 1mo ago First seen · 672 lines · 78 tokens per session scan A 0ef6a87d42e7
shipkit-semantic-qa is a skill published in the GitHub repository stefan-stepzero/shipkit (1 stars, last pushed 3mo ago), licensed MIT. It adds 78 tokens to every session and 7,344 once invoked, about $0.0003 per session on Opus 5.5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.
Other skills, from other repositories
visual-qa
Runs rigorous visual QA across web, terminal, and paginated surfaces: an Apple HIG-based checklist, paired light/dark captures at phone and desktop widths, screenshot evidence, and a per-item PASS/FAIL verdict. Use for any UI build or change, or when asked whether a page, component, or TUI looks right or follows the…
remove-ai-slops
Removes AI-generated code smells from branch changes or an explicit file list behind regression tests. Use when the user asks to clean up, deslop, or remove AI-slop patterns from recent changes.
visual-regression
Setup visual regression testing with Storybook stories, configuration, and CI/CD workflows. Supports Chromatic, Percy, BackstopJS. Auto-invoke when user says "set up visual regression", "add Chromatic tests", "add screenshot testing", or "set up Percy".
test-coverage-improver
Analyze test coverage gaps and generate tests to improve coverage. Use when improving test coverage, finding untested code, or writing missing tests.
browser-test-loop
Use to actually verify a web change in a real browser instead of assuming it works — click through a flow, catch console errors, check a form submits, confirm a fix on the page. Trigger when a UI change needs proof, a bug only reproduces in the browser, or the user asks to test, check, or screenshot the running app.
Vizra ADK Evaluation Framework
Test and evaluate AI agents with automated evaluations, assertions, and LLM-as-a-Judge patterns.