leaderboard: Skill for Claude Code

Benchmark discovery for the Steel leaderboard

Surface candidate benchmark results for maintainer review. The skill drives the deterministic arxiv discovery script in this repo, applies project-specific filtering rules, and renders a standardised markdown summary that a maintainer can scan and selectively add to src/data/<slug>.json.

This skill never modifies leaderboard data on its own. The output is review material; merging is always a human step. The discovery issue exists so candidates accumulate in one inbox rather than getting lost across notebooks and tabs.

Automated weekly sweep (Atlas research → deductive judge → deterministic PR)

A scheduled GitHub Actions pipeline runs this discovery loop weekly so a maintainer doesn't have to. It is Propose-Dispose: the LLM stages only propose — they have no tools and never touch git — and a deterministic apply step validates, re-ranks, and opens the PR a human merges. The workflows live in .github/workflows/:

• discovery-scheduler.yml — cron (Mondays ~07:27 UTC) that dispatches the sweep via gh workflow run with since_days=9 (weekly + 2-day margin). Kill-switch: set repo variable ENABLE_DISCOVERY=false to pause without editing YAML.
• discover-results.yml — the sweep itself, four jobs:
  1. arxiv-baseline — free, deterministic pnpm run discover → .discovery/run.json (§1 below).
  2. discover — read-only. pnpm run discover:research (the Atlas driver at src/scripts/discover-research.ts: Exa search + Steel browser fetch, z.ai glm-5.2) produces cited, structured candidates in .discovery/atlas-candidates.json (plus atlas-report.md, atlas-trace.jsonl, atlas-errors.json). Then pnpm run discover:judge — a deductive generateObject pass at src/scripts/discover-judge.ts, with no tools and no fetches — reads the candidates + the FULL current leaderboard rows + CONTRIBUTING.md and emits one declarative proposal per benchmark to .discovery/proposals/<dataKey>-<week>.json (adds + dismissals, each with a reason and a category).
  3. apply — the ONLY job with contents:write + pull-requests:write. pnpm run discover:apply --write (deterministic TypeScript at src/scripts/discover-apply.ts, no LLM) hard-gates each add (src/data/schema.json, finite score, dedup, freshness), does a soft SSRF-guarded evidence re-fetch, stably re-ranks (existing rows keep their order; new rows slot in by score), writes src/data/<dataKey>.json + README.md, runs update-readme/lint/build, and opens one PR per leaderboard with per-row reasoning for every add AND dismissal.
  4. notify — on success posts PR links (or "no fresh results") to the rolling issue; on any failure — including a silent gate/build failure where survivors were proposed but no PR opened — it opens a discovery-alert issue. The alert label is deliberately separate from the rolling tracker's discovery label so the tracker's find-or-create never misroutes a comment into an alert.