maplibre-running-evals

Running Promptfoo Evals

Every eval here runs two roles over provider APIs: a generator, the model under test, and a judge, the model that grades each llm-rubric assertion. evals/README.md covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.

1. Set up and budget the run

Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.

• Confirm a generator key and a judge key are set — presence only, never the value: echo "generator: ${GROQ_API_KEY:+set}" and echo "judge: ${GOOGLE_API_KEY:+set}". If either is missing, stop and report; do not extrapolate a result from a run you did not make.
• Both roles are pinned: the generator in evals/prompts/lib/providers.yaml, the judge in package.json's eval:graded script. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both.
• Send every recorded run through npm run eval:graded, the only place --grader, --delay, and -j are set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass --providers either: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — evals/README.md, "Writing eval prompts" has the detail. --providers echo is the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but every llm-rubric still calls the judge, so even a probe spends judge calls.
• Learn the provider's limits before launching: read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the x-ratelimit-* headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a 429 arrives — retry-after gives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.