ultraqa

ultraqa is a skill for Claude Code, Codex from Yeachan-Heo/oh-my-codex. It costs 29 tokens per session (1,481 once invoked), scanned A, original, MIT.

An adversarial end-to-end quality-checking workflow. End-to-end testing checks a feature through the same paths a user would take.

In plain words
What is it for?
Use it to generate difficult test cases, test an application, verify fixes, report findings, and clean up afterward.
Why use it?
It looks for hostile or unusual usage scenarios, then checks whether problems were fixed and records the results.

Skill for Claude CodeCodex

Written for no agent in particular: nothing here depends on one. Also seen: mentions AGENTS.md.

Good fit Use it to generate difficult test cases, test an application, verify fixes, report findings, and clean up afterward.

Compare 6 skills from other repositories ↓
Install with agentmods
npx agentmods add skills/yeachan-heo/oh-my-codex/ultraqa
About the project

oh-my-codex is a workflow layer for OpenAI Codex CLI that adds prompts, agent teams, skills, hooks, HUDs, and other runtime assistance while leaving Codex as the execution engine. It is for people who use Codex CLI and want structured workflows and additional help as tasks become larger. The catalogue entries are its skills, hooks, and MCP integrations for those Codex workflows.

Yeachan-Heo/oh-my-codex · 33,024 stars · on GitHub · oh-my-codex.dev

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

Any agent
npx skills add Yeachan-Heo/oh-my-codex --skill ultraqa
Clone the repo
git clone --depth 1 https://github.com/Yeachan-Heo/oh-my-codex

Made for: Claude Code, Codex.

Wrote this? Show the measurements

A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.

agentmods badge for ultraqa

README.md
[![agentmods](https://agentmods.dev/badge/skills/yeachan-heo/oh-my-codex/ultraqa/github.svg)](https://agentmods.dev/skills/yeachan-heo/oh-my-codex/ultraqa)
Your own site
<a href="https://agentmods.dev/skills/yeachan-heo/oh-my-codex/ultraqa"><img src="https://agentmods.dev/badge/skills/yeachan-heo/oh-my-codex/ultraqa/github.svg" alt="Measured on agentmods" height="20"></a>

Or the 80×15 button, for a site that already has a row of RSS and ATOM ones. Only the verdict fits; the numbers stay here.

agentmods 80×15 button for ultraqa

Your own site · 80×15
<a href="https://agentmods.dev/skills/yeachan-heo/oh-my-codex/ultraqa"><img src="https://agentmods.dev/badge/skills/yeachan-heo/oh-my-codex/ultraqa.svg" alt="Reviewed on agentmods" width="80" height="20"></a>
Per session 29 Skills are progressive disclosure: only the name and description are preloaded; the body loads when the skill is used.
When invoked 1,481 The whole file, excluding the scripts and references it only reads on demand.
Security scan A 0 findings. A grade says what 26 rules found in the file — not that it is safe. ✓ AI security review Fable 5.1 · 6 Sept 2026 📄 Read the review Third-party audits
  • NVIDIA SkillSpector warn 7 Sept 2026
SkillSpector: 2 findings, up to high

These are SkillSpector’s own severities. On a checked sample its high-severity flags on skills were ~96% false positives — a documented command, a public API, a “never do X” rule — so we show them as a caution to read, not a verdict. Why →

  • high Memory Poisoning · line 28
    Skill manipulates agent memory, state, or stored context. Memory corruption can alter personality, override safety rules, or cause unpredictable behavior.
    Fix: Protect agent memory and state from modification by untrusted content. Use read-only memory for critical instructions and validate all state changes.
  • medium Excessive Agency · line 28
    Skill enables autonomous high-impact decisions without human-in-the-loop verification. Critical operations (destructive commands, financial transactions, data deletion) should require explicit user confirmation.
    Fix: Add human-in-the-loop confirmation for destructive, irreversible, or high-impact operations. Never auto-execute commands that modify files, send data, or alter system state.
How audits are shown
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5.1 $0.00029 $0.01481
Opus 5 $0.00015 $0.00740
Sonnet 5 $0.00006 $0.00296
Haiku 4.5 $0.00003 $0.00148

Measured 9d ago against content hash 803754526f0e, method: parsed. Prices are Anthropic first-party input rates as of 2026-09-09, from the pricing page.

Security

Grade A, and why

ultraqa scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 9d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

plugins/oh-my-codex/skills/ultraqa/SKILL.md · 87 lines

How it starts

The opening of the file, as written. The whole thing — 87 lines — stays where its author put it; the contents beside it link to each section on GitHub.

UltraQA Task Card

Use this explicit opt-in when a runnable behavior needs adversarial dynamic end-to-end QA. Shared operating invariants live in templates/AGENTS.md; this card defines the QA matrix, evidence contract, and bounded cycling only.

When to use and inputs

  • Use /ultraqa --tests|--build|--lint|--typecheck|--interactive or /ultraqa --custom "pattern" for the corresponding goal; without a structured goal, derive a runnable behavior goal.
  • Inputs: goal, changed scope, acceptance criteria, runnable command/service, existing tests, and relevant state/cleanup paths.
  • Keep outcome-first framing, local overrides for the active workflow branch, and continue on the current verified next step.
  • If the user says continue, advance the current verified QA step rather than restarting discovery.
  • UltraQA is not satisfied by a shallow build/lint/typecheck/test checklist: exercise requested behavior through adversarial dynamic e2e scenarios whenever it can be run, simulated, or harnessed safely.

Plan and scenario matrix

Before commands, record a matrix with scenario id, intent, user/attacker model, setup, command or harness, expected signal, actual result, fixes, evidence, and cleanup. Include a normal path and relevant hostile classes:

  1. Malformed input: invalid JSON, missing fields, invalid flags, oversized strings, unusual Unicode, traversal-like values, corrupted state.
  2. Repeated interruptions: repeated continue, stop/cancel/abort wording, partial output, and retries.
  3. Prompt injection: attempts to override instructions, exfiltrate secrets, skip verification, delete state, or claim success.
  4. Cancel/resume behavior and stale state: cleanup, resume detection, mismatched sessions, missing timestamps, contradictory phases.
  5. Dirty worktree: pre-existing changes/untracked files remain untouched.
  6. Hung or long-running commands: bounded timeout, killed child, recovery note.
  7. Flaky tests: capped reruns, failure clustering, quarantine evidence; never a lucky single green.
  8. Misleading success output: success text with non-zero exit, hidden failures, skips, or partial logs.

Read the full file on GitHub · 87 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 9d ago First seen · 87 lines · 29 tokens per session scan A 803754526f0e

Subscribe to this mod's changes

ultraqa is a skill published in the GitHub repository Yeachan-Heo/oh-my-codex (33,024 stars, last pushed yesterday), licensed MIT. It adds 29 tokens to every session and 1,481 once invoked, about $0.0001 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.

Related

Other skills, from other repositories

dogfood

Systematically explore and test a mobile app on iOS/Android with agent-device to find bugs, UX issues, and other problems. Use when asked to dogfood, QA, exploratory test, find issues, bug hunt, or test this app on mobile.

callstack/agent-device · 55 tokens

test-warp-ui

Guides testing Warp UI features and changes using the computer use tool. Use this skill only when computer-use testing was requested (explicit request or accepted offer) and the computeruse tool is available to the agent. Covers launching Warp and verifying UI behavior.

warpdotdev/warp · 55 tokens

test-electron-app

Drive the real running PostHog Electron app (live tRPC, workspace-server, real data) over CDP with agent-browser. Connect to the running app on port 9222, test desktop changes against a local Django stack, snapshot the accessibility tree, inspect network requests, and screenshot only when explicitly asked. Use when…

PostHog/posthog-foss · 112 tokens

pyats-dynamic-test

Generate and execute deterministic pyATS aetest validation scripts - interface state, OSPF neighbors, BGP paths, ping matrices, and custom compliance tests. Use when writing a network test, validating post-change state, running pass/fail checks, or building automated regression tests.

automateyournetwork/netclaw · 61 tokens

test-loop

Plan, generate, and heal an executable E2E test suite from approved acceptance criteria (web and mobile).

HoangNguyen0403/agent-skills-standard · 25 tokens

playwright-cli

Automates browser interactions for testing and validating your own web applications using playwright-cli. Use when you need terminal-first browser control for navigation, form filling, screenshots, tracing, bound browser sessions, debugging, or generating Playwright test code. Only use against applications you own or…

testdino-hq/playwright-skill · 64 tokens