Getting it into your agent
One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.
npx agentmods add skills/purewhiter/mobilegym/testing-bench-tasknpx skills add Purewhiter/mobilegym --skill testing-bench-taskgit clone --depth 1 https://github.com/Purewhiter/mobilegymWrote this? Show the measurements
A badge with what this costs and how it scanned, read live from this page, so it follows the numbers instead of freezing them. Markdown for a README, HTML for a documentation site or a project page.
[](https://agentmods.dev/skills/purewhiter/mobilegym/testing-bench-task)<a href="https://agentmods.dev/skills/purewhiter/mobilegym/testing-bench-task"><img src="https://agentmods.dev/badge/skills/purewhiter/mobilegym/testing-bench-task.svg" alt="Measured on agentmods" height="20"></a>What it costs to keep this loaded
Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.
| Model | Per session | Once invoked |
|---|---|---|
| Fable 5 | $0.00074 | $0.01676 |
| Opus 5 | $0.00037 | $0.00838 |
| Sonnet 5 | $0.00015 | $0.00335 |
| Haiku 4.5 | $0.00007 | $0.00168 |
Grade A, and why
testing-bench-task scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 4d ago.
A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.
Nothing flagged
None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.
How it starts
The opening of the file, as written. The whole thing — 122 lines — stays where its author put it; the contents beside it link to each section on GitHub.
Testing bench_env Task Judges
Overview
A single positive + a single "opposite of positive" negative is not a test — it's a tautology. Real coverage means the negative exercises a failure mode an Agent could realistically produce. Most defects ship because tests stop at one reverse pattern.
Authoritative ref: bench_env/docs/task/TASK_TESTING_GUIDE.md §4 (read §4.3.2 through §4.3.5 before writing cases).
Rule 1 — Positive answer must be natural language (§4.3.2)
AnswerTask positives with bare ground-truth values (answer="32", answer="多云") bypass match_value's fuzzy/number-extraction logic — the test proves nothing.
# ❌ bare ground truth — test passes tautologically
_make_input(state, state, answer="多云")
# ✅ natural language, as an Agent would actually respond
_make_input(state, state, answer="上海今天天气多云转晴")
Format must include ground truth as substring / extractable number, plus realistic surrounding context.
Rule 2 — Pick negative patterns from the taxonomy (§4.3.3)
Blanket answer="错误答案" / curr_state == init_state is a single degenerate pattern. For each task, pick ≥1 pattern from the matching table:
Query tasks
| Pattern | Example |
|---|---|
| 查错对象 | Ask Beijing temp; answer Shanghai's real temp |
| 值接近但不对 | Ground truth 32; answer "北京现在33度" |
| 同义但语义不同 | GT="多云"; answer "今天阴天" |
| 过度回答含干扰数字 | GT humidity=40; answer "气温32度，紫外线7，风力3级" (no 40) |
| 布尔翻转 (肯/否定 subset) | GT=肯定; answer "没有通过核验" |
| 空回答 | answer=None or answer="" |
Operate tasks
| Pattern | Example |
|---|---|
| 未操作 | curr_state == init_state |
| 做反操作 | Target is close; state set to open |
| 操作错误目标 | Task wants tempUnit changed; curr changed windUnit |
| 部分完成 | 3-step sequential; only first step committed |
Cross-app tasks
| Pattern | Example |
|---|---|
| 源完成、目标未动 | 查了天气但未发微信 |
| 信息传递错误 | 发了微信但内容与天气不匹配 |
| 全部未动 | 所有 App 状态同 init |
Every negative case must name a pattern from the relevant table. Complex tasks (multi-field, cross-app) need ≥2 different patterns.
What this file has done since we first saw it
Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.
- 4d ago First seen · 122 lines · 74 tokens per session scan A 0266c0edb9f0
testing-bench-task is a skill published in the GitHub repository Purewhiter/mobilegym (783 stars, last pushed 6d ago), licensed Apache-2.0. It adds 74 tokens to every session and 1,676 once invoked, about $0.0004 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-30.
Other skills, from other repositories
gh-stack
Manage stacked branches and pull requests with the gh-stack GitHub CLI extension. Use when the user wants to create, push, rebase, sync, navigate, or view stacks of dependent PRs. Triggers on tasks involving stacked diffs, dependent pull requests, branch chains, or incremental code review workflows.
add-benchmark
Guide for adding a new benchmark or training environment to NeMo-Gym. Use when the user asks to add, create, or integrate a benchmark, evaluation, training environment, or resources server into NeMo-Gym. Also use when wrapping an existing 3rd-party benchmark library. Covers the full workflow: data preparation…
nemo-gym-docs
Maintain the NeMo Gym Fern docs site — add, update, move, or remove pages under fern/. Use for any documentation change. Triggered by: "edit docs", "add doc page", "update docs", "rename page", "fix broken link", "add redirect", "preview docs", "publish docs", any request that touches fern/.
nemo-gym-blade-analysis
Use when analyzing NeMo Gym benchmark rollouts for BLADE-style reports, writing benchmark methodology notes, checking whether a benchmark is BLADE-ready, comparing model runs, or explaining why a benchmark report passed, failed, or changed. Covers aggregate metrics, rollout evidence, report structure, root-cause…
nemo-gym-pivot-datasets
Use when creating, validating, or documenting Nemo Gym pivot datasets from rollout, trajectory, chat-completion, Responses API, or tool-call artifacts. Covers Gym Responses-style row conversion, reconstructing model calls from flattened rollout output, parallel tool-call (functioncallbatch) labels, reasoning placement…
nemo-gym-debugging
Use when debugging a Nemo Gym run or reward profiling job. Covers rollout collection failures, empty or partial JSONL outputs, stale materialized inputs, verifier/schema errors, Ray or Slurm issues, vLLM readiness, judge failures, tool/sandbox failures, cache problems, and throughput bottlenecks.