What is ticket-test-runner?

A workflow that turns a Jira ticket into a browser-based test using Playwright, a tool for controlling web browsers. It can also score AI or text output with RetriEval and record the result in Jira.

Use it to verify or perform quality checks for Jira tickets that contain the required test instructions.

It removes the need to manually read test instructions, run them, judge eligible output, and update the ticket.

ticket-test-runner — Skill for Claude Code and Codex by hcarrillo001

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent

npx skills add hcarrillo001/retrieval-mcp --skill ticket-test-runner

Clone the repo

git clone --depth 1 https://github.com/hcarrillo001/retrieval-mcp

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

README.md

[![agentmods](https://agentmods.dev/badge/skills/hcarrillo001/retrieval-mcp/ticket-test-runner/github.svg)](https://agentmods.dev/skills/hcarrillo001/retrieval-mcp/ticket-test-runner)

Your own site

<a href="https://agentmods.dev/skills/hcarrillo001/retrieval-mcp/ticket-test-runner"><img src="https://agentmods.dev/badge/skills/hcarrillo001/retrieval-mcp/ticket-test-runner/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

Your own site · 80×15

<a href="https://agentmods.dev/skills/hcarrillo001/retrieval-mcp/ticket-test-runner"><img src="https://agentmods.dev/badge/skills/hcarrillo001/retrieval-mcp/ticket-test-runner.svg" alt="Reviewed on agentmods" width="80" height="20"></a>

Per session 139 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.

When invoked 1,080 The whole file, excluding the scripts and references it only reads on demand.

Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe.

Origin original No closer match found in the catalogue.

Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

Model	Per session	Once invoked
Fable 5.1	$0.00139	$0.01080
Opus 5	$0.00069	$0.00540
Sonnet 5	$0.00028	$0.00216
Haiku 4.5	$0.00014	$0.00108

Measured 8d ago against content hash 19f66dafe023, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-09, from the pricing page.

Security

Grade A, and why

ticket-test-runner scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 8d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

ticket-test-runner/SKILL.md · 96 lines

How it starts

The opening of the file, as written. The whole thing — 96 lines — stays where its author put it; the contents beside it link to each section on GitHub.

Ticket Test Runner

Turn a Jira ticket into a tested, scored, auto-updated ticket. Given one ticket key, this skill reads the ticket, runs the test it describes in a real browser, scores any AI/text output, and posts the verdict back to Jira.

Required connectors

Jira MCP — Atlassian Rovo MCP (Cloud) or mcp-atlassian (Server/DC). Needs read and write (comment + transition).
Playwright MCP (Microsoft) — browser automation.
RetriEval MCP — output scoring. Only needed for tickets with an eval block.

Before starting, confirm these are connected. If one is missing, tell the user exactly which to connect and stop — do not fake any step.

Input

A single ticket key, e.g. PROJ-123. The ticket description must contain a test block (see references/ticket-format.md). If it doesn't, post a comment listing the required fields and stop — never guess the test.

Workflow

Read the ticket. Fetch the issue by key via the Jira MCP. Find the fenced ```test block in the description and parse: url, steps, expected, and the optional eval section. Read references/ticket-format.md for the exact spec before parsing.
Run the test with Playwright. Open url, execute each step in order (type / click / select / wait as written). Capture the observed result — the text the steps point at — plus a screenshot, and a DOM snapshot if a step fails. Mark each step pass/fail against expected. If a selector is missing or a step is ambiguous, mark it BLOCKED and ask; do not assume pass.
Score the output (only if the ticket has an eval section). Build one case — input = the query text, actual_output = the captured answer, expected_output / retrieval_context from the ticket — and call RetriEval evaluate_case with the listed metrics (default: faithfulness, answer_relevancy) at the given threshold. Pass generator_model / judge_model if the ticket names them. Record each score and pass/fail.

Read the full file on GitHub · 96 lines

Files

What ships with it

2 files beside SKILL.md in the same directory: the scripts, references and assets a skill reads on demand. Not counted in the per-session cost; read them before you install if any of them is executable.

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

8d ago First seen · 96 lines · 139 tokens per session scan A 19f66dafe023

Subscribe to this mod's changes

ticket-test-runner is a skill published in the GitHub repository hcarrillo001/retrieval-mcp (0 stars, last pushed 13d ago), licensed Apache-2.0. It adds 139 tokens to every session and 1,080 once invoked, about $0.0007 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.

Browser automation E2E testing Testing

browse

Drive a real browser through Aside: open a page, read it, click through a flow, take screenshots, check console errors. (gstack).

garrytan/gstack · 32 tokens

playwright-cli

A command-line tool for controlling Chromium, Firefox, and WebKit browsers, including navigation, page interaction, screenshots, PDFs, and recorded actions.

UnicomAI/wanwu · 76 tokens

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

anthropics/skills · 35 tokens

create-verification-skill

Create a repo-local verification skill and exhaustive feature map for driving a real app through its UI, CLI, or API. Use for "create a verification skill", "make a verify skill for this repo", or "document how agents can verify this app".

get-bb/bb · 57 tokens

browser-qa

A browser-based quality check for deployed web pages and user flows. It uses browser automation to test rendering, navigation, forms, interactions, responsive behaviour, and accessibility-related issues.

hashgraph-online/awesome-codex-plugins · 58 tokens

test-electron-app

Drive the real running PostHog Electron app (live tRPC, workspace-server, real data) over CDP with agent-browser. Connect to the running app on port 9222, test desktop changes against a local Django stack, snapshot the accessibility tree, inspect network requests, and screenshot only when explicitly asked. Use when…

PostHog/posthog-foss · 112 tokens