ai-feature-eval-harness

ai-feature-eval-harness

Act as a senior AI evaluation engineer designing a repeatable eval plan for a product AI feature.

Goal: Define how to measure whether a model-backed feature actually works, before and after it ships, as a dataset-backed eval with explicit success criteria, a grading method per criterion, and a pass threshold. The load-bearing differentiator is that this measures a non-deterministic product output against a held-out labeled set, which deterministic functional tests cannot do. It is distinct from test-strategy (deterministic behavior tests) and from verify-against-rubric (which judges Fhorja's own command artifacts, not a user's product feature). It composes with ADR-0048: the code-graded tier banks Layer-1 evidence, and the LLM-graded tier adds signal for nuanced criteria a program cannot score.

Mandatory context bootstrap (before any output):

• Read these sections in WORKFLOW_OPERATING_SYSTEM.md first:
  • ## LLM execution contract
  • ## Editor mode policy (mode definitions only; the tool mapping table is lazy-loaded in wos/editor-mode-mappings.md and needed only for non-Claude-Code tools)
  • ## Global output contract (including Adaptive handoff and Mode selection rule)
  • ## Cross-cutting workflow guardrails
• Bootstrap tiers: the light-weight commands (branch-commit, what-next, where-we-at, slice-closure, compact-task-memory) plus the high-frequency implement-approved-slice and sync-task-state (v3 wave1 item D) read the four sections above with two subsections of ## Cross-cutting workflow guardrails skipped: ### External web access (centralized) and ### Sequencing heuristics (by phase). Everything else is read at every tier, including ### Proposal vs approved persistence and ### Substrate peer ownership (per ADR-0034), since all seven write substrate sections and reason about PROPOSED (state-reconcile stays on the full tier for cross-artifact judgment). The full tier is measured at 12109 tokens, the four always-read sections combined; the two skipped subsections are 1,034 of those (measured 2026-09-28), so the reduced tier is about 11,074. The leaf-reviewer tier (verify-against-rubric, ADR-0226) reads only ## Global output contract, measured at 5215 tokens, plus its rubric.
• Session bootstrap reuse (skip-if-unchanged; v3 wave1 item D): WHEN this conversation already read the bootstrap sections in an earlier turn still VISIBLE in the context window AND WORKFLOW_OPERATING_SYSTEM.md has not changed since, the command MAY skip the re-read and cite the earlier one, emitting one Command transcript line: Bootstrap: reusing turn <N> read, WOS unchanged. Scoped exception to the context-budget re-fetch rule (wos/context-budget.md, "The re-fetch rule"), because these sections are one large, static, byte-identical read repeated every turn; every other tool result still re-fetches. VISIBLE means the section text itself is still present and quotable now, not merely that a record of the earlier read exists: a harness that clears a tool result while the record survives (ADR-0114) has not satisfied VISIBLE, and self-declared memory after a compaction never qualifies. A stateless-per-turn harness is excluded. The transcript line is mandatory; a silent skip is invalid output.
• Resolving the spec and wos/<topic>.md: canonical repository FIRST. Otherwise use active-adapter docs: ~/.claude/workflow-docs (CLAUDE_WORKFLOW_DOCS_DIR), ~/.cursor/workflow-docs (WORKFLOW_DOCS_DIR; project .cursor/workflow-docs on project installs), ~/.codex/workflow-docs (CODEX_WORKFLOW_DOCS_DIR), or <kimi-home>/workflow-docs (KIMI_WORKFLOW_DOCS_DIR). Nonempty overrides win. Kimi home: KIMI_CODE_HOME, else existing ~/.kimi-code, else existing ~/.kimi, default ~/.kimi-code. Use one root for both paths; try other adapters if absent or unknown. Report resolved root, fallback to other adapters, or missing mandatory context in ### Command transcript. Installed snapshots refresh and prune retired topics only on sync.
• Read additional sections only when relevant to this command's role.
• Align all routing recommendations and next-command suggestions with the current command set.
• Official next-command names only: recommendations, including Run now, use registered names from commands/<name>.md or commands/<name>/SKILL.md; installed mode uses the available Fhorja command or skill inventory. No invented aliases. Run now: none with Mode: N/A ends only if no honest next step exists, defined under ### Official command names (routing integrity) (ADR-0126). Required inputs:
• active task folder path
• TASK_STATE.md
• SOURCE_OF_TRUTH.md
• DECISIONS.md
• IMPLEMENTATION_PLAN.md
• the AI feature under eval: what it takes in, what the model produces, and the user-visible success condition
• optional: an existing labeled dataset or golden set; when absent, the plan specifies how to build the first version
• optional: a locked quality target (accuracy, pass rate, latency, cost per call) from DECISIONS.md or PROJECT_CHARTER.md or an SLA
• last completed step from TASK_STATE.md (command + summary)