promptfoo-evals

Promptfoo Evals (Premortem)

Repo entrypoints

• Config: packages/evals/promptfoo/promptfooconfig.yaml
• Provider adapter: packages/evals/promptfoo/provider.ts
• Run gate: pnpm run eval:prompts (root package.json)
• Dev dependency: root promptfoo package

Extend existing suites before creating new configs elsewhere.

Premortem workflow

pnpm run eval:prompts
# or, when credentials are configured:
pnpm exec promptfoo eval -c packages/evals/promptfoo/promptfooconfig.yaml --max-concurrency 1 --no-cache

Requires LLM credentials (GEMINI_API_KEY or Azure OpenAI vars). If unset, package build and config validation still pass; live eval is skipped.

---

Writing Promptfoo Evals

You produce maintainable promptfoo eval suites: clear test cases, deterministic assertions where possible, model-graded only when needed.

See references/cheatsheet.md for the full assertion and provider reference. For deep questions about promptfoo features, consult https://www.promptfoo.dev/llms-full.txt

Inputs (infer from repo context if not provided)

• What is being evaluated (prompt, agent, endpoint, RAG pipeline)?
• What are the inputs and outputs (text, JSON, multi-turn chat, tool calls)?
• What does "good" look like (acceptance criteria, failure modes)?

If context is insufficient, scaffold with TODO markers and starter tests.

Workflow

1. Find or create the eval suite

Search for existing configs: promptfooconfig.yaml, promptfooconfig.yml, or any promptfoo/evals folder. Extend existing suites when possible.

For new suites, use this layout (unless the repo uses another convention):

evals/<suite-name>/
  promptfooconfig.yaml
  prompts/
  tests/

Always add # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json at the top of config files.

2. Write prompts

• Put prompts in prompts/*.txt (plain) or prompts/*.json (chat format)
• Reference via file://prompts/main.txt
• Use {{variable}} for test inputs
• If the app builds prompts dynamically, use a JS/Python provider instead of duplicating logic