10-hardness-gate

A Cursor rule for judging whether a benchmark task is genuinely difficult. It defines a hardness gate, a pre-build check, and a diagnostic probe for testing task difficulty.

In plain words
What is it for?
It is for designing benchmark tasks, checking whether their constraints make the intended solution hard to discover, and diagnosing results that were too easy.
Why use it?
It helps task authors avoid benchmarks that look difficult but allow a solver to find the best answer cheaply. The rule also describes limits on how certain the assessment can be.

Cursor rule for Cursor

Install

Getting it into your agent

One page per mod, every tool's command on it. A separate URL per tool would split the same page into five that compete with each other.

agentmods
npx agentmods add rules/xclaw-bot/benchmark-task-authoring/10-hardness-gate
Clone the repo
git clone --depth 1 https://github.com/Xclaw-bot/benchmark-task-authoring

Made for: Cursor.

Per session 1,669 This file is loaded in full into every session.
When invoked 1,669 The same file — it is already loaded in full.
Security scan A 0 findings. Scan, not verified.
Origin original No closer match found in the catalogue.
Token cost

What it costs to keep this loaded

Counted locally with the o200k_base tokenizer, which is exact for GPT models; Claude uses its own tokenizer and its counts differ. Treat this as one consistent yardstick across the catalogue rather than a bill. Prices are per million input tokens.

ModelPer sessionOnce invoked
Fable 5 $0.01669 $0.01669
Opus 5 $0.00834 $0.00834
Sonnet 5 $0.00334 $0.00334
Haiku 4.5 $0.00167 $0.00167

Measured 2d ago against content hash 9c2479953cc1, method: parsed. Prices are Anthropic first-party input rates as of 2026-08-30, from the pricing page.

Security

Grade A, and why

10-hardness-gate scanned grade A with 0 findings against 26 rules in 11 categories — prompt injection, anti-refusal, data exfiltration, privilege escalation, supply chain, agent snooping, system-prompt leakage, SSRF and excessive agency — measured 2d ago.

A static scan of the body, not an audit. Every finding is printed with the line that produced it so you can judge whether it matters here. A mod is markdown that instructs an agent; that is exactly why what it instructs is worth reading.

Nothing flagged

None of the 26 patterns this scan looks for appear in this file: no shell pipes, no recursive deletes, no credential paths, no hidden text, no instruction-override or anti-refusal phrasing, no agent-config snooping. That is not a guarantee, it is the absence of the things that are checkable.

.cursor/rules/10-hardness-gate.mdc · 124 lines

How it starts

The opening of the file, as written. The whole thing — 124 lines — stays where its author put it; the contents beside it link to each section on GitHub.


description: The hardness law, its pre-build check and the strong-probe protocol. Load when designing a task, judging whether a design is hard enough, or diagnosing a pass@ result that came back too easy. alwaysApply: false

The hardness gate

Canonical source: ../hardness-laws.md. Derived over seven printed-optimum tasks (four measured pass@5 clears, three withdrawn on probe). Read its "Honest limits" section before treating any of this as certainty — at n=5 against a 60% bar, a genuinely 70%-fail design still misses the gate ~16% of the time.

The law

Ship only if, with every stated rule held correctly, a competent solver still cannot cheaply construct the optimum. That requires ≥2 load-bearing beliefs that are misreadable, narrowing, public-blind, unrepairable by restart, and silent under the printed optimum, plus one stated resource constraint that makes the bulk/uniform architecture illegal.

Whether a belief is stated in one sentence or entailed across sections is irrelevant — that axis is dead. State everything. A stated rule is worth exactly what its measured wrong-belief rate says.

Why those conditions and not others

The agent's cheapest winning strategy is transcribe → simulate → over-enumerate → check against N. Each condition closes one leg of it:

  • A transcribed simulator enforces per-step legality for free. So any fact entailed by step legality is worth zero — the simulator rejects violations without ever deriving the fact. Difficulty can only live in facts needed to emit a candidate: whole-plan or terminal properties, or the identity/layout of the objects acted on. Constructive, not checkable.
  • A printed optimum is a free belief oracle on sealed data. cost == N runs inside the graded program with no feedback channel. A wrong belief that overshoots, undershoots, wedges, crashes or exhausts a resource is an alarm, and one retry repairs it. Only beliefs whose wrong plans self-report cost == N, all checks pass survive.
  • Restart repairs every over-permissive slip. It cannot cross an action-space boundary. Measured: one slot's over-permissive seal-blind was restarted around 12/12; its narrowing freeze-blind killed 9/12 and was unrepairable.

Read the full file on GitHub · 124 lines

Changes

What this file has done since we first saw it

Hashed on every crawl. A supply-chain change to an agent config is a question of when, not whether, so the history is kept rather than the latest state alone.

  1. 2d ago First seen · 124 lines · 1,669 tokens per session scan A 9c2479953cc1

Subscribe to this mod's changes

10-hardness-gate is a cursor rule published in the GitHub repository Xclaw-bot/benchmark-task-authoring (2 stars, last pushed 17d ago), licensed MIT. It adds 1,669 tokens to every session, about $0.0083 per session on Opus 5. A static security scan graded it A with 0 findings. No closer match exists in the catalogue, so it is treated as the original; first seen 2026-08-31.