agentmods.dev

shipkit-semantic-qa

shipkit-semantic-qa is a skill for Claude Code, Codex from stefan-stepzero/shipkit. It costs 78 tokens per session (7,344 once invoked), scanned A, original, MIT.

A quality-checking workflow that tests an API or user interface against criteria you define. For interfaces, it can judge screenshots; for backends, it can judge responses.

In plain words
What is it for?
Use it to define test inputs and quality rules, generate and run test scripts, check API outputs, or review whether a built interface matches the intended design.
Why use it?
It finds missing, incorrect, or visually inconsistent results after implementation and before final verification.

Skill for Claude CodeCodex

1 1mo ago A scan 78 tokens original MIT
View the source file stefan-stepzero/shipkit
Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add skills/stefan-stepzero/shipkit/shipkit-semantic-qa
Any agent
npx skills add stefan-stepzero/shipkit --skill shipkit-semantic-qa
Clone the repo
git clone --depth 1 https://github.com/stefan-stepzero/shipkit

Made for: Claude Code, Codex.

Per session 78 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 7,344 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.00078 $0.07344
Opus 5 $0.00039 $0.03672
Sonnet 5 $0.00016 $0.01469
Haiku 4.5 $0.00008 $0.00734

Measured 3d ago against content hash 0ef6a87d42e7, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

shipkit-semantic-qa scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 3d ago.

The scan reads SKILL.md. This mod also ships 7 executable files (tools/fidelity/_common.py, tools/fidelity/_declared.py, tools/fidelity/_smoke/coach/leaderboard.py, …), listed below but not scanned — reading those needs a real analyzer, not pattern matching.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

install/skills/shipkit-semantic-qa/SKILL.md · 672 lines

How it starts

The opening of the file, as written. The whole thing — 672 lines — stays where its author put it; the contents beside it link to each section on GitHub.

shipkit-semantic-qa - Semantic Quality Assurance

Purpose: Define test inputs and quality criteria, generate test scripts, run them, and let Claude semantically judge outputs (API responses or UI screenshots) against human-defined criteria.

Pattern: One skill, one loop — Setup → Run → Judge. Two suite types: backend (API/LLM pipeline) and frontend (visual components).

When to Invoke

User triggers:

  • "Semantic QA", "Set up QA", "Quality check"
  • "Visual QA", "Screenshot QA", "Check my UI"
  • "Judge outputs", "Run QA suite", "Check quality"
  • "Set up quality criteria", "Define test inputs"

Workflow position:

  • After features are implemented (something to test)
  • Before verify/preflight (catches quality issues early)
  • Can run standalone against any API or UI

Prerequisites

Required: None (Setup mode creates everything)

Helpful:

  • .shipkit/stack.json — Tech stack informs script generation
  • .shipkit/specs/ — Acceptance criteria can seed quality criteria
  • Playwright installed (for frontend suites only)

Process

Completion Tracking

In --full mode (all 3 phases sequential), create tasks at the start:

  • TaskCreate: "Setup: Define criteria + generate test script"
  • TaskCreate: "Run: Execute tests + verify output count"
  • TaskCreate: "Judge: Evaluate ALL outputs against ALL criteria"
  • TaskCreate: "Write judgment.md + judgment.json"

TaskUpdate each task to in_progress when starting it, completed when done.

In Judge mode with 5+ outputs, create one task per output to prevent partial evaluation. Do NOT present judgment summary until all output evaluations are complete and files are written.

Step 0: Mode Detection

Determine mode from arguments and state:

Explicit flag provided:

  • --setup → Setup mode
  • --run → Run mode
  • --judge → Judge mode
  • --full → Setup (if needed) → Run → Judge
  • --fidelity → Fidelity mode — score a built app against captured intent (completeness + essence), emit a fidelity scorecard. See Step 5. Add a second suite/arm to compare two builds under one rubric.

Read the full file on GitHub · 672 lines

Files

What ships with it

19 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 3d ago First seen · 672 lines · 78 tokens per session scan A 0ef6a87d42e7

Subscribe to this mod's changes

shipkit-semantic-qa is a skill published in the GitHub repository stefan-stepzero/shipkit (1 stars, last pushed 1mo ago), licensed MIT. It adds 78 tokens to every session and 7,344 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Testingai-toolsanthropicclaudeclaude-codeclaude-skillsdeveloper-tools
Related

Other skills, from other repositories

opencli-sitemap-author

Use when creating or maintaining OpenCLI site sitemaps: agent-facing navigation, page-state, action, workflow, API-reference, pitfall, and fallback knowledge for a website. Use after browser exploration discovers durable site context, when a sitemap is stale, or when promoting local site knowledge into the repo.

jackwener/OpenCLI · 67 tokens

golden-rss

Use when testing the rss golden build.

yusufkaraaslan/Skill_Seekers · 12 tokens

omh-code-review

This is a Hermes-native code-review workflow skill.

rlaope/oh-my-hermes · 46 tokens

redteam-web-detail-pack

Routing and boundary guidance for authorized general web application security testing. Use as a web testing router when the attack surface should be dispatched to more specific web vulnerability skills.

Netw0rkNoob/VulnClaw · 38 tokens

android-pentest

安卓应用渗透测试 — APK分析、Hook、自动化测试、运行态驱动、签名恢复、抓包分析.

Netw0rkNoob/VulnClaw · 32 tokens

Vizra ADK Memory System

Implement persistent memory, session context, and vector memory (RAG) for AI agents.

vizra-ai/vizra-adk · 24 tokens