mobilegym: Skill for Claude Code

Writing bench_env Task Judges

Overview

A judge decides whether the Agent completed the task. Two failure modes dominate:

• Soundness hole — an obviously wrong path is judged passed=True (keyword hit, unstable trace field).
• Completeness hole — a reasonable completion path is judged passed=False (bound to specific wording or UI step).

Neither is caught by type checks — both are caught by applying the CRUD model rigorously.

Authoritative refs:

• bench_env/docs/task/TASK_AUTHORING_GUIDE.md — task design + CRUD judge model (read §2 in full)
• bench_env/docs/task/TASK_CODE_SPEC.md — code rules (file responsibilities, naming, defensive-coding ban, time APIs, CriteriaTask)
• bench_env/docs/REFERENCE.md — JudgeInput / JudgeResult field & expected_changes path syntax lookup
• bench_env/docs/task/GROUNDED_MODE.md — only if task uses answer_fields / grounded eval

Step 1 — Classify the task as CRUD

Before writing any check, name the operation out loud:

Operation	Triggering verbs	Required check strategy
Create (增)	添加/创建/发送/收藏/点赞/新建	diff: init vs current, match in new items
Delete (删)	删除/移除/取消/取关/下架	diff: target id in init_ids - curr_ids
Modify (改)	修改/切换/改名/设置为/拨动	lookup in current; use init to resolve identity
Query (查)	告诉我/是多少/哪个/什么时候	read from init, compare to input.answer

Each CRUD type has ONE correct strategy — not a choice. Using lookup-on-current for Delete/Create creates false positives (sampler bug → passes as Agent action).

Step 2 — Survey the App module for reusable abstractions

Before writing any judge, Read app.py end-to-end with the Read tool (split into sections only if the file exceeds the tool's limit). Scan every def line plus its docstring / leading comment, plus any module-level helper functions above the class. Do not substitute grep for this. Grep silently misses:

• module-level helpers (_note_text, _pick_keywords_from_note) that sit above the class
• @staticmethod / @property / @classmethod methods that your prefix guess won't anticipate
• helpers whose names don't follow the shape you grepped for (latest_note_by_title is a find-by-name primitive but doesn't start with find_; visible_notes is a filtered view but doesn't start with filtered_)
• _xxx private methods that contain logic you could extract