docs-eval-planning

Planning a documentation eval

A documentation eval measures a page, not an agent. The prompt sends a user's request plus the page's url, and the checks say whether an agent that read the page produced working code. A gap in the page counts as a failure.

That framing is the whole difficulty. The failure modes are collected in references/flakiness.md. Read it before designing checks, not after.

What this skill produces

A plan. It does not write PROMPT.md, EVAL.ts, or a seed. Implementation and local scoring are a separate job.

The shape it takes

Model a new eval on build-docs-003-api-keys-guide or later.

build-docs-NNN-<subject>, named after the doc rather than the feature. The id is permanent. The published results series is keyed on it, so renaming breaks history. So does renaming a check.

evals/docs/build-docs-NNN-<subject>/
  PROMPT.md      frontmatter and the task the agent sees
  EVAL.ts        the scorer
  README.md      the design rationale, addressed to the next editor
  <helper>.ts    flat beside EVAL.ts, never in a subdirectory
  local/         the seed workspace

CONTRIBUTING.md makes the README.md optional. For a page under test it is required: it is the only place the stripped-word list and the constraints that must not be edited away live. Its sections:

• What this eval measures. The page is the subject, not the agent.
• Do not reintroduce the vocabulary. The stripped words, listed.
• The seed carries the contract. What the seed fixes, and what each fixed thing buys and costs.
• Do not drop the positive controls. Which checks pass for an agent that built nothing, and which ones make them mean something.
• The guide has to actually be read.
• What this eval does not score. Each entry with its reason. An unmeasured rule from the prompt is named here.

EVAL.ts is orchestration. Import named check* functions, assemble one flat array, return { passed: checks.every(c => c.passed), checks }, and wrap it in a try/catch whose catch returns a single self-named failure check. Split the implementations into modules and keep the full check list declared in EVAL.ts. Comment why the phase order is what it is.