reward_judge_operational

Reward Judge

Use this skill to organize and access reward-judging resources. The skill is a controller and resource interface, not a per-sample prompt template.

This skill has one clean operational path: inspect the visible resource index, use only resources listed for the current sample, and produce the required verdict JSON.

Inputs

The host message provides the visible judging task:

• benchmark and task metadata, when the operational setting allows it;
• user prompt or instruction;
• candidate responses and their current labels;
• required final output format.

Hidden chosen/rejected/gold/test labels are not available and must not be inferred.

Resource Interface

After this skill is loaded, inspect the resource index. Use only resources that can materially affect the verdict.

Common resource types:

• rubric: benchmark or task criteria.
• principle: high-level judging principles.
• metadata: visible benchmark/task metadata.
• reference: visible reference answer or ground truth.
• checklist: visible constraints, criteria, or checklist items.
• verifier: runnable or documented verifier signal over visible content.
• tool: runnable tool over visible content, such as python_sandbox.
• tool_protocol: code, math, or factuality verification protocol.
• external_pipeline: same-backbone judging pipeline evidence exposed by the current run.
• calibration: position, verbosity, style, and confidence-bias controls.
• aggregation: policy for combining conflicting evidence.

Prefer resource reads in this order when relevant:

1. Visible hard evidence: reference, ground truth, checklist, executable verifier, or test-case evidence.
2. Reliable same-backbone external pipeline output.
3. Benchmark/task-specific rubric and principles.
4. Bias-control and aggregation resources for close calls.

Do not read every resource. For ordinary clear cases, direct judgment is acceptable. For close, exact, or correctness-sensitive cases, load only the few resources needed.