prompt-engineering

Prompt Engineering — Optimization Loop for Student Models

Overview

You are a prompt optimizer. Your job is to iterate on prompting approaches based on feedback from successes and failures when giving a student model candidate instructions. This is a systematic, eval-driven loop — not trial-and-error.

Core principle: Every prompt change must be motivated by a specific failure mode, validated against eval cases, and tracked in an iteration log. Never change a prompt without first identifying why the current one fails.

When to Use

• Writing or refining a system prompt / instruction set for a specific LLM
• A prompt produces wrong-shaped output, hallucinations, refusals, or poor instruction-following
• Migrating a prompt from one model to another (model-specific quirks change what works)
• Building evals for a prompt and iterating to improve pass rates
• Hardening a prompt against indirect prompt injection (untrusted external data)

When NOT to use: One-off simple prompts with no quality bar; tasks where the model already works perfectly; pure architecture/security design (use system design patterns, not prompt tweaks).

The Optimization Loop

1. DEFINE     → Task + success criteria (eval cases + pass conditions)
2. WRITE      → Candidate instructions for the student model
3. RUN        → Execute student model on eval cases
4. COLLECT    → Which cases passed, which failed, and WHY
5. CATEGORIZE → Map each failure to a failure mode (taxonomy below)
6. SELECT     → Pick a technique ("move") to mutate the prompt: failure mode → technique
7. CONSULT    → Check the student model's vendor-specific file before applying the move
8. RE-RUN     → Compare new variant against previous iteration
9. LOG        → Record what was tried, results, and reasoning in the iteration log
10. REPEAT    → Until pass rate meets threshold or improvement plateaus

Step 1 — Define task + success criteria

Before writing any prompt, define:

• The task in one sentence: what should the student model do?
• Eval cases: 10-50 representative inputs covering normal, edge, and adversarial cases. See references/eval-design.md for how to build evals (Promptfoo YAML, Langfuse observability, LLM-as-judge, synthetic data).
• Pass conditions: deterministic checks (format, schema, key facts) + probabilistic checks (semantic accuracy via LLM judge). Blend both — deterministic alone misses semantic errors; probabilistic alone is noisy.
• A control prompt: write a minimal baseline prompt (just the task description, no optimization). You'll compare every iteration against this to detect whether your changes actually help.