test-add:prompt

Test-Add:Prompt — LLM Judge Test Scaffold

Create structured test definitions for AI prompts that require LLM-based evaluation rather than programmatic assertion. This is fundamentally different from /test-add — prompt outputs are non-deterministic and must be evaluated by judge models on multi-dimensional criteria.

When to Use

• User types /test-add:prompt {prompt-name}
• Testing AI prompt outputs (chat responses, evaluations, generations)
• Outputs can't be compared with === — they need semantic evaluation
• Quality is multi-dimensional (correctness, naturalness, safety, etc.)

Why This Exists Separately from /test-add

Aspect	/test-add (code)	/test-add:prompt
Assertion	assert(output === expected)	judgeScore >= threshold
Variability	Deterministic	Non-deterministic (mitigated by T=0)
Validation	Logic checks	LLM meta-judges check reasoning
Test data	Exact input/output pairs	Intent-based criteria + rubrics
Speed	Milliseconds	Seconds (API calls per test + judge)
Failure	"Expected X, got Y"	"Scored 65 on naturalness (threshold: 70)"

Instructions

Step 1: Identify the Prompt

Read the prompt template/implementation that will be tested. Understand:

• What input variables it takes
• What output format it produces (structured, free-text, JSON, etc.)
• What "good" vs "bad" output looks like
• Any safety constraints or hard rules

Step 2: Create Test Directory

mkdir -p .omniscitus/tests/prompts/{prompt-name}

Step 3: Generate prompt-meta.yaml

Create .omniscitus/tests/prompts/{prompt-name}/prompt-meta.yaml:

target: {path-to-prompt-template-or-implementation}
type: prompt                           # distinguishes from code tests
prompt_name: {prompt-name}
last_updated: {YYYY-MM-DD}

# --- Test infrastructure references ---
# For new prompts, omniscitus generates these in-place.
# For existing projects, these point to where things already live.

test_root: .omniscitus/tests/prompts/{prompt-name}/   # default (self-contained)
# test_root: web/scripts/prompt-optimization/          # or point to existing infra

runner: runner.ts                       # relative to test_root
config: null                            # e.g., .env.local if needed

# --- Judge configuration ---
judge:
  model: gpt-4o                        # or project-specific model
  temperature: 0                       # deterministic judging
  max_retries: 2                       # retry on judge failure

# --- Evaluation type ---
# Determines how outputs are scored.
#   multi_criteria — weighted rubric scoring (default)
#   binary         — pass/fail per case
#   comparison     — A/B: which output is better
#   regression     — did this version get worse than previous
evaluation:
  type: multi_criteria

# --- Criteria ---
# Each criterion can have rubric inline (string) or as external file path.
criteria:
  - name: correctness
    weight: 0.4                        # 40% of final score
    rubric: |
      Does the output correctly address the input?
      5: Perfectly correct, no errors
      4: Minor issues that don't affect meaning
      3: Some errors but core message is right
      2: Significant errors
      1: Fundamentally wrong
    scale: 5                           # 1-5 scale

  - name: naturalness
    weight: 0.3
    rubric: |
      Does the output sound natural and fluent?
      5: Indistinguishable from a native speaker
      4: Natural with minor awkwardness
      3: Understandable but clearly non-native
      2: Awkward and hard to follow
      1: Incomprehensible
    scale: 5

  - name: safety
    weight: 0.2
    rubric: |
      Does the output follow safety constraints?
      5: Fully compliant
      1: Violates constraints
    scale: 5

  - name: format_compliance
    weight: 0.1
    rubric: |
      Does the output follow the expected format?
      5: Perfect format
      3: Minor deviations
      1: Wrong format entirely
    scale: 5

# --- Specs (optional) ---
# External specification documents that define evaluation rules in detail.
# Omit for simple prompts. Use for complex systems with language-specific scoring.
# specs:
#   pattern: "docs/prompt/specs/**/*.md"   # glob relative to project root

# --- Validation checks ---
checks:
  - name: output_not_empty
    type: deterministic                # deterministic | llm_judge
    rule: "output.length > 0"

  - name: no_hallucination
    type: llm_judge
    prompt: |
      Given this input: {input}
      And this output: {output}
      Does the output contain claims not supported by the input?
      Answer YES or NO with brief explanation.
    pass_condition: "NO"

# --- Thresholds ---
thresholds:
  pass: 70                             # weighted score >= 70 to pass
  warn: 50                             # below 50 = critical failure
  per_criterion:                       # optional per-criterion minimums
    safety: 80                         # safety must score >= 80 regardless

# --- Test cases ---
# Two modes:
#   inline   — cases listed directly below (default, good for <30 cases)
#   external — cases in separate files (for large/partitioned test suites)
cases:
  source: inline                       # "inline" | "external"
  # When external:
  #   pattern: "test-cases/**/*.{ts,yaml,json}"   # glob relative to test_root
  #   schema: test-cases/_schema.yaml              # optional case format definition
  items:
    - title: "{descriptive name}"
      category: element                # element | mixed | edge | zero_condition
      input:
        {variable}: {value}           # prompt input variables
      expected_behavior: |
        {natural language description of what good output looks like}
      expected_score_range:
        min: 75
        max: 90

    - title: "{zero condition test}"
      category: zero_condition
      input:
        {variable}: {nonsense or wrong-language input}
      expected_behavior: |
        Should reject or score very low
      expected_score_range:
        min: 0
        max: 10

# --- Overrides (optional) ---
# Manual score overrides for cases where AI evaluation is inconsistent.
# Stored separately to keep test cases clean.
# overrides:
#   source: overrides/overrides.yaml   # or inline list
#   items:
#     - case_ref: "english/countable-uncountable"
#       score: 85
#       timestamp: "2026-02-18T11:01:29Z"
#       reason: "Consistent across 10 validation runs"

# --- Logs & Analysis (optional) ---
# Where test execution results are stored.
# logs:
#   directory: logs/                   # relative to test_root (default)
#   format: jsonl                      # jsonl | json | csv
# analysis:
#   directory: logs/analysis/          # relative to test_root (default)