prompt-evaluator

Prompt evaluator

Evaluate observable output behavior instead of judging whether a prompt looks sophisticated.

Define the evaluation target

Record:

• Prompt or skill version.
• Provider, model, snapshot, surface, and generation settings.
• Intended users and task distribution.
• Required properties and costly failure modes.
• Non-determinism budget: number of runs per case when consistency matters.

Do not compare outputs produced with undocumented setting differences.

Build the case set

Include:

1. Typical cases representing common use.
2. Boundary cases at format, length, language, or capability limits.
3. Ambiguous cases where the prompt should ask, assume, or decline in a defined way.
4. Adversarial cases with conflicting or untrusted instructions.
5. Preservation cases for facts, identity, layout, quotations, or references.
6. Regression cases for every previously observed material failure.

Use the test-case template. Keep fixtures free of secrets and personal data.

Choose graders

Prefer deterministic checks for properties that can be measured exactly:

• Valid JSON or required headings.
• Field presence and allowed values.
• Length bounds.
• Exact preservation of protected text.
• Absence of forbidden additions.
• Source or citation presence.

Use a rubric grader for qualities such as clarity, faithfulness, composition, or tone. Define anchored score levels and require evidence from the output. Human review remains necessary for high-stakes claims, identity, creative quality, and ambiguous failures.

Load the evaluation rubric for scoring.

Run a controlled comparison

1. Freeze inputs and settings.
2. Generate enough repetitions to expose material variance.
3. Blind labels when human preference could be biased.
4. Score each requirement independently.
5. Compare failure rates, not only average style preference.
6. Inspect regressions by case category.
7. Change one prompt dimension and repeat.