next-evals-oss: Skill for Claude Code

Adding a model to nextjs.org/evals

The deliverable is a landed run, not a config

A model addition is done when the run has happened, agent-results.json contains its numbers, and a PR against the site repo carries that file. It is not done when the experiment config exists.

export-results only exports experiments that have results, so a registered-but-unrun model does not reach the board at all — the PR looks complete and changes nothing a reader can see. The README's retention policy names this state and rejects it: "a staging post, not a destination — land the run and tier the model in the same PR, or drop the config."

So do not stop after step 2 below and hand back a "ready to run, just needs the go-ahead" PR. The run costs real money (roughly $50–250 for a Fable/Opus-tier pair at list) and takes 30–90 minutes of wall clock. Spend it, then report what it cost. Stop early only when a step is genuinely blocked — pnpm preflight says a credential is missing, or the gateway refuses the model id.

Steps

1. Set up and verify credentials

pnpm install --frozen-lockfile
pnpm sync-evals 071a2343c509751585cd9f77ae66e8c30daf2ea8   # the SHA .github/workflows/eval-cache-check.yml pins
pnpm preflight

Sandbox auth needs either VERCEL_OIDC_TOKEN or all three of VERCEL_TOKEN + VERCEL_TEAM_ID + VERCEL_PROJECT_ID. One or two of the triple is worse than none — it silently falls back to OIDC and dies on Could not get credentials from OIDC context. In a Vercel Devbox the token is usually in the environment and the two IDs are in the injected instructions; put the missing ones in .env.local (gitignored).

Do not proceed on a red preflight.

2. Register the experiment pair

Confirm the model id and the effort rung against the live gateway before writing either into a config. Catalogs disagree with the API and with each other; the gateway is the source of truth.

# Id: an id the gateway does not serve answers 404 model_not_found, so a 200 here is
# real resolution rather than a silent fallback to some default model.
curl -s -o /dev/null -w '%{http_code}\n' https://ai-gateway.vercel.sh/v1/messages \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{"model":"<id>","max_tokens":16,"messages":[{"role":"user","content":"ok"}]}'

# Effort: a bad rung returns 400 enumerating the real set. Worth more than any catalog.
curl -s https://ai-gateway.vercel.sh/v1/chat/completions \
  -H "Authorization: Bearer $AI_GATEWAY_API_KEY" -H 'Content-Type: application/json' \
  -d '{"model":"<provider/id>","reasoning_effort":"<rung>",
       "messages":[{"role":"user","content":"ok"}],"max_tokens":2000}'