modern-web-guidance-src: Skill for Claude Code

Stage 3: Evaluating guidance for a use case (Needs evals)

This is the third of three stages in creating guidance:

1. Stage 1: Identifying use cases for a feature
2. Stage 2: Authoring guidance for a use case
3. Stage 3: Evaluating guidance for a use case (you are here)

What the eval agent sees vs real-world agents

Real-world coding agents see only guide.md — retrieved via the Skills CLI when a developer asks for help. Every other file in a guide directory is eval infrastructure.

The eval harness runs a coding agent in an isolated workspace initialized with a target base application (daily-grind or devtools-times). The agent receives the prompt from targets/<base_app>/task.md and (in guided mode) access to guide.md via the Skills CLI. The harness then runs targets/<base_app>/grader.ts against the agent's modified application.

None of the following are ever seen by real-world coding agents:

File	Role in eval pipeline
expectations.md	Natural-language spec of assertions that must hold when guide.md is followed
demo.html	Standalone reference implementation authored by the SME
targets/<base_app>/task.md	A concrete, single-bullet user prompt tailored to <base_app>, formatted as a single - line with no YAML frontmatter
targets/<base_app>/patches/*-solution.patch	Multi-agent golden solution diffs (claude, codex, jetski) that must score 100% on grader.ts
targets/<base_app>/patches/zero-passrate.patch	Baseline anti-pattern diff that must score 0% on grader.ts
targets/<base_app>/grader.ts	Playwright + static AST test suite run against the eval agent's output

Writing expectations.md

Write a natural-language, bulleted list of assertions that must be true if an agent implements guide.md correctly.

• 1:1 with grader tests — Each bullet becomes one test() block in grader.ts (for use-case guides). Write one bullet per assertion; do not combine multiple checks into a single bullet.
• Plain declarative phrasing — Write each bullet as a plain statement of what is true of a correct implementation (e.g., "The dialog closes when the Escape key is pressed"). Omit The implementation MUST… boilerplate; imperative directives belong in guide.md.
• Concrete, testable criteria — Every expectation must be verifiable via static AST analysis (Linkedom, CSSOMNom, ts-morph) or Playwright browser execution.
• Scoped to this use case — Only include expectations that apply to the specific use case being graded and are documented in guide.md.
• Avoid untestable conditional expectations — Avoid "If the implementation uses X, then Y" expectations when valid solutions may omit X entirely; if a golden solution patch has zero instances of X, the grader cannot assert Y without writing a vacuous guard.
• No external links — The grader generator cannot resolve them.
• Avoid over-constraining — Describe required outcomes, API contracts, or structural invariants rather than prescribing incidental implementation details (specific variable names, helper names, or file paths).