harness-bench-fast: Instructions file for Codex

Repository Instructions

Benchmark Result Tables

When reporting итоговую таблицу benchmark runs for this repository, use this table format by default:

harness	model	passed	score	steps	tokens

Rules:

• Include only completed runs with exactly 391 tasks (current task-set v0.16.0) unless the user asks for partials.
• Sort rows by score descending.
• Use passed as <passed>/391 and score as a percentage with one decimal place.
• Do not mix task-set versions in one table. Earlier runs are not comparable: v0.14.0 scored 371 tasks, and v0.16.0 corrected defects across the 391 of v0.15.0 without changing their number. If more than one version must appear, split the table and name the task-set version in each.
• Do not include artifact links or artifact paths in the main table unless explicitly requested.
• Use the human-readable harness and model names, not only the JSON filename.
• Include steps and tokens from the run JSON when present.
• If a run artifact does not contain steps or token metrics, show 0 and note that 0 means the metric is absent from the artifact, not that nothing was spent.

Scoring Rules

• A task interrupted by the per-task timeout or an agent hang counts as a normal fail (kept in the full denominator, no "partial/interrupted" note).
• Transient infrastructure errors (network failures, HTTP 5xx / 429 / 529 Overloaded, connection resets, gateway timeouts) are not model failures. A task that died on such an error must be rerun before the run is reported; the retried result replaces the errored one. In-flight auto-retries of transient errors by runners are allowed and do not need to be disclosed per task.

Model Identification

• When recording or reporting any benchmark run, always save not only the exact model (id/version/build) but also the reasoning level used, when the model or API supports one (e.g. reasoning_effort, thinking budget, effort presets like low/medium/high). If no level was set explicitly, record it as default — reasoning level materially changes both scores and token spend, so a result without it is not reproducible.