Narraitor: Skill for Claude Code

Narraitor AI quality discipline

1. Purpose

Prevent the most seductive failure in this project: judging stochastic AI behavior by a single sample. Generation quality claims need the same rigor as code claims — defined coverage, deliberate failure-path exercise, recorded results.

2. When to use

Any change to prompts, templates, context assembly, generation config, parsing/normalization, or any claim about narrative/choice/ending quality.

3. When not to use

• The generation pipeline is erroring (500s, timeouts, parse crashes) → narraitor-debugging-playbook.
• Registering/structuring a template → narraitor-prompt-template-governance (the lifecycle); this skill supplies its eval gate.

4. Inputs required

The exact change (template diff, config diff); a live dev server with a working provider key; at least two contrasting worlds and characters (create them if needed — that IS the work).

5. Procedure

The core axioms:

1. One good generation ≠ a reliable prompt. Gemini at temperature 0.7 varies per call; a good sample proves possibility, not tendency.
2. A passing unit test ≠ shippable UX. Unit tests pin parsing and template assembly; they cannot pin prose quality, pacing, or coherence.
3. Model output is not ground truth. It is confidently wrong exactly when it reads most fluent — verify claims the narrative makes about world/character state against the stores.
4. Coherence is an arc property. A turn can read well while contradicting the session — judge across a multi-turn arc (does it respect established lore facts, character attributes, prior decisions, inventory?).
5. "Reads better" is a hypothesis. It becomes a result only via the protocol below.

The evaluation protocol (minimum bar for any behavior claim). This section is the SINGLE HOME of the matrix minimums — sibling skills cite it rather than restate numbers; if another file disagrees with this section, this section wins:

1. Fix the variable: change ONE thing (one template, one config value) per evaluation.
2. Coverage matrix: >= 2 contrasting worlds (different genre/tone) x >= 2 characters
   (one established with history, one fresh) — 3+ generations per cell.
3. Arc check: for narrative changes, play >= 3 consecutive turns in one cell;
   verify continuity against loreStore facts and prior segments.
4. Failure drill: exercise the non-happy path at least once —
   empty/malformed response handling (the parse/normalize layer), a slow response,
   and a missing/invalid key. MECHANISM, by failure type — they need different tools.
   There is NO in-app mock toggle; mock at the network boundary or below:
   - Slow / error / rate-limit / intermittent: intercept the route. In Playwright,
     tests/visual/utils/mockApi.ts already routes the AI endpoints and takes
     per-endpoint delays (narrativeDelayMs / choicesDelayMs / endingDelayMs); for a
     non-200 or an intermittent failure, add a `page.route` handler that fulfills with
     the status you want. In Jest, mock the src/lib/api wrapper the component calls.
   - Truly malformed/empty response BODY (the parse-error path — e.g. the item-image
     20%-parse-failure class): exercise at the parse layer with a unit fixture (a
     `Response` whose `.json()` throws). A routed mock still returns well-formed bytes,
     so it cannot reproduce this class.
   - Missing/invalid key: omit or corrupt the provider key (see diagnostics-and-tooling).
     Routes fall back on their own when no key resolves, so a keyless dev environment
     is itself a useful drill.
   Timeouts are layered: 30s server-side on generate/choices, 120s client-side in aiFetch.
5. Record: for each cell, verdict + one representative excerpt + what you compared against.
   Template: `.claude/skills/narraitor-prompt-template-governance/templates/eval-log.md`.
6. Verdict language: "improved on the evaluated matrix" — never "better" unqualified.