eval-first-development

Eval-First Development

1 · Purpose

Guarantee that every later change to a prompt, model, or pipeline is judged by a fixed measuring stick, so "better" is a number, not a feeling.

2 · When to use

• Before writing or optimizing any prompt, choosing any model, or tuning any retrieval pipeline whose output quality matters.
• When a team is debating two prompts/models with no score to cite.

Do not use for throwaway scripts and one-off data pulls. Use it the moment the output will be shown to anyone as "working".

3 · Inputs

• A spec with a metric and gate (spec-first-ai-feature), or permission to define the metric here.
• At least 20 real or realistic inputs. If fewer exist, generate the rest synthetically and label them synthetic: true.

4 · Procedure

1. Create the golden-set file (e.g. golden.jsonl): one JSON object per case with input, expected, and tags (the failure classes you anticipate). Start with 20 cases; 50 is better. Include the hard cases you hope never occur.
2. Write the scorer as code, a function score(output, expected) -> dict that returns per-field or per-case results. Exact match where possible; rubric-graded LLM-as-judge only where correctness is fuzzy, and calibrate the judge against 10 human-graded cases before trusting it.
3. Add the score report: one line per case plus an aggregate, written to evals/results/<timestamp>.json. A score you cannot diff is a story.
4. Run the scorer against the current system (or a trivial baseline) and record the baseline score in the spec file.
5. Freeze the golden set. Changes to it are commits with a message, never silent edits, the measuring stick must not move while the system is being measured.
6. Wire the scorer so one command runs it (script, make target, or notebook cell). If running the eval takes more than one command, it will not get run.
7. Only now touch the thing being improved, prompt, model, chunking, tools. Every subsequent change reports: score before, score after, one-line reason.